Sync in Progress on both @wavi and @mospherePro on Windows

cconstab commented 2 years ago

Describe the bug Both apps get stuck on Sync in Progress

To Reproduce Steps to reproduce the behavior:

First install on Windows latest versions
Then try to run apps
My Atsigns are on Canary BUT this only started happening last week.. Mobile Apps seem fine

Expected behavior Should be able to Sync !

Screenshots

Windows (please complete the following information):

PC Windows 11 Were you using an @‎application when the bug was found?
- [e.g. @‎wavi, @‎tmosphere]

Additional context Unclear to me why this started happening last week.. My Secondary has been up for days

@info
data:{"version":"3.0.17+c3.0.18a","uptimeAsWords":"16 days 0 hours 25 minutes 3 seconds","features":[{"name":"noop:","status":"Beta","description":"The No-Op verb simply does nothing for the requested number of milliseconds. The requested number of milliseconds may not be greater than 5000. Upon completion, the noop verb sends 'ok' as a response to the client.","syntax":"^noop:(?<delayMillis>\\d+)$"},{"name":"info:","status":"Beta","description":"The Info verb returns some information about the server including uptime and some info about available features. ","syntax":"^info(:brief)?$"}]}
@

Only thing I cxan think of is the Windows version got updated ? @sachins-geekyants ?

cconstab commented 2 years ago

And the logs on the client show the problem (running @mospherePro

The reason this happened last week is that @ kevin was deleted last week and then reinstated but has not been enabled yet. This is arguably a SDK issue and a bug in @ mospherePro.

Would really like to nail and learn from this issue as much as we can @gkc / @murali-shris / @VJag on for the Arch call tomorrow.. @sachins-geekyants see what you think!

lutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@kevin"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-06 15:50:14.352466|AtLookup|AtLookup.findSecondary received atsign: kevin
flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@kevin"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-06 15:50:14.361467|AtLookup|AtLookup.findSecondary got answer: 1619b378-f1bb-5f63-b7bc-fe628380f7e7.canary.atsign.zone:2364 and closing connection
flutter: FINER|2022-06-06 15:50:14.399499|AtLookup|AtLookup.findSecondary received atsign: kevin
flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@kevin"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-06 15:50:14.445507|AtLookup|AtLookup.findSecondary received atsign: kevin
flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@kevin"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-06 15:50:14.458508|AtLookup|AtLookup.findSecondary got answer: 1619b378-f1bb-5f63-b7bc-fe628380f7e7.canary.atsign.zone:2364 and closing connection
flutter: FINER|2022-06-06 15:50:14.459507|AtLookup|AtLookup.findSecondary got answer: null and closing connection

cconstab commented 2 years ago

Outstanding oddness is why does the Mobile app work still ?

murali-shris commented 2 years ago

@sachins-geekyants is creating a new release for windows with latest at_client and at_client_mobile dependency

sachins-geekyants commented 2 years ago

@cconstab can you try running windows app from v1.0.7.42 tag.

cconstab commented 2 years ago

In tried but Windows keeps deleting the binary

cconstab commented 2 years ago

@sachins-geekyants @murali-shris

At this stage I can no longer log in on Windows...

The key is correct

cconstab@canary-01:~$ docker service logs -f --tail 10  79b6d83f-5026-5fda-8299-5a0704bd2416_secondary
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vw6ijlu27k4l@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 10:01:58.429933|AtSecondaryServer|Closing AccessLog HiveBox
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vw6ijlu27k4l@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vw6ijlu27k4l@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 10:01:58.430038|AtSecondaryServer|Closing NotificationKeyStore HiveBox
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vw6ijlu27k4l@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vw6ijlu27k4l@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 10:01:58.441840|AtSecondaryServer|Closing Main key store HiveBox
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vw6ijlu27k4l@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vw6ijlu27k4l@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 10:01:58.453267|AtSecondaryServer|Stopping scheduled tasks
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vw6ijlu27k4l@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vw6ijlu27k4l@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 10:01:58.453366|SecondaryServerBootStrapper|secondaryServerInstance.stop() completed, and isRunning is false - exiting with status 0
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vw6ijlu27k4l@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.dnl5sqasfskr@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:26:34.062957|AtSecondaryServer|Closing AccessLog HiveBox
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.dnl5sqasfskr@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.dnl5sqasfskr@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:26:34.063285|AtSecondaryServer|Closing NotificationKeyStore HiveBox
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.dnl5sqasfskr@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.dnl5sqasfskr@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:26:34.075020|AtSecondaryServer|Closing Main key store HiveBox
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.dnl5sqasfskr@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.dnl5sqasfskr@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:26:34.086468|AtSecondaryServer|Stopping scheduled tasks
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.dnl5sqasfskr@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.dnl5sqasfskr@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:26:34.086531|SecondaryServerBootStrapper|secondaryServerInstance.stop() completed, and isRunning is false - exiting with status 0
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.dnl5sqasfskr@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.mxubvokkmqa6@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:47:21.252314|AtSecondaryServer|Closing AccessLog HiveBox
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.mxubvokkmqa6@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.mxubvokkmqa6@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:47:21.252452|AtSecondaryServer|Closing NotificationKeyStore HiveBox
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.mxubvokkmqa6@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.mxubvokkmqa6@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:47:21.261778|AtSecondaryServer|Closing Main key store HiveBox
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.mxubvokkmqa6@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.mxubvokkmqa6@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:47:21.271074|AtSecondaryServer|Stopping scheduled tasks
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.mxubvokkmqa6@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.mxubvokkmqa6@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:47:21.271142|SecondaryServerBootStrapper|secondaryServerInstance.stop() completed, and isRunning is false - exiting with status 0
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.mxubvokkmqa6@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.sczdoblr02lu@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:38:03.175071|AtSecondaryServer|Closing AccessLog HiveBox
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.sczdoblr02lu@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.sczdoblr02lu@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:38:03.175130|AtSecondaryServer|Closing NotificationKeyStore HiveBox
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.sczdoblr02lu@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.sczdoblr02lu@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:38:03.184325|AtSecondaryServer|Closing Main key store HiveBox
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.sczdoblr02lu@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.sczdoblr02lu@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:38:03.195339|AtSecondaryServer|Stopping scheduled tasks
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.sczdoblr02lu@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.sczdoblr02lu@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-05-21 21:38:03.195386|SecondaryServerBootStrapper|secondaryServerInstance.stop() completed, and isRunning is false - exiting with status 0
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.sczdoblr02lu@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:40:52.761459|InboundListener|RCVD: [_977307f7-8a5d-463b-a4e4-613af49e8656] noop:0
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:52.761505|AtSecondaryServer|inside _executeVerbCallBack: noop:0
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:52.761571|DefaultVerbExecutor|verb handler found : NoOpVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:52.761705|NoOpVerbHandler|Verb : noop  Response: Response{_data: ok, _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:40:52.761743|InboundConnectionImpl|SENT: [_977307f7-8a5d-463b-a4e4-613af49e8656] data:ok
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:58.220422|AtSecondaryServer|In _listen - clientSocket.peerCertificate : null
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:40:58.220534|InboundConnectionImpl|SENT: [_6308d506-e67e-4037-9b5f-d1f2ca6cda99] @
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:40:58.307712|InboundListener|RCVD: [_6308d506-e67e-4037-9b5f-d1f2ca6cda99] scan
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:58.307775|AtSecondaryServer|inside _executeVerbCallBack: scan
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:58.307824|DefaultVerbExecutor|verb handler found : ScanVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:58.317916|ScanVerbHandler|response.data : [custom_about_me.wavi@colin, custom_butterflies.wavi@colin, custom_callsign.wavi@colin, custom_catch-up_and_book_a_meeting.wavi@colin, custom_github.wavi@colin, custom_jeeping.wavi@colin, custom_my_first_professional_computer.wavi@colin, custom_proverb.wavi@colin, custom_quote_of_the_day.wavi@colin, discord.wavi@colin, email.wavi@colin, facebook.wavi@colin, field_order_of_self.wavi@colin, firstname.wavi@colin, followers_of_self.at_follows.wavi@colin, following_by_self.at_follows.wavi@colin, image.wavi@colin, instagram.wavi@colin, lastname.wavi@colin, linkedin.wavi@colin, location.wavi@colin, locationnickname.wavi@colin, phone.wavi@colin, privateaccount.wavi@colin, profilepic.buzz@colin, pronoun.wavi@colin, publickey, publickey@colin, signing_publickey@colin, theme.wavi@colin, theme_color.wavi@colin, twitter.wavi@colin, youtube.wavi@colin]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:58.317999|ScanVerbHandler|keysArray : [custom_about_me.wavi@colin, custom_butterflies.wavi@colin, custom_callsign.wavi@colin, custom_catch-up_and_book_a_meeting.wavi@colin, custom_github.wavi@colin, custom_jeeping.wavi@colin, custom_my_first_professional_computer.wavi@colin, custom_proverb.wavi@colin, custom_quote_of_the_day.wavi@colin, discord.wavi@colin, email.wavi@colin, facebook.wavi@colin, field_order_of_self.wavi@colin, firstname.wavi@colin, followers_of_self.at_follows.wavi@colin, following_by_self.at_follows.wavi@colin, image.wavi@colin, instagram.wavi@colin, lastname.wavi@colin, linkedin.wavi@colin, location.wavi@colin, locationnickname.wavi@colin, phone.wavi@colin, privateaccount.wavi@colin, profilepic.buzz@colin, pronoun.wavi@colin, publickey, publickey@colin, signing_publickey@colin, theme.wavi@colin, theme_color.wavi@colin, twitter.wavi@colin, youtube.wavi@colin], 33
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:58.318053|ScanVerbHandler|Verb : scan  Response: Response{_data: ["custom_about_me.wavi@colin","custom_butterflies.wavi@colin","custom_callsign.wavi@colin","custom_catch-up_and_book_a_meeting.wavi@colin","custom_github.wavi@colin","custom_jeeping.wavi@colin","custom_my_first_professional_computer.wavi@colin","custom_proverb.wavi@colin","custom_quote_of_the_day.wavi@colin","discord.wavi@colin","email.wavi@colin","facebook.wavi@colin","field_order_of_self.wavi@colin","firstname.wavi@colin","followers_of_self.at_follows.wavi@colin","following_by_self.at_follows.wavi@colin","image.wavi@colin","instagram.wavi@colin","lastname.wavi@colin","linkedin.wavi@colin","location.wavi@colin","locationnickname.wavi@colin","phone.wavi@colin","privateaccount.wavi@colin","profilepic.buzz@colin","pronoun.wavi@colin","publickey","publickey@colin","signing_publickey@colin","theme.wavi@colin","theme_color.wavi@colin","twitter.wavi@colin","youtube.wavi@colin"], _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:40:58.318080|InboundConnectionImpl|SENT: [_6308d506-e67e-4037-9b5f-d1f2ca6cda99] data:["custom_about_me.wavi@colin","custom_butterflies.wavi@colin","custom_callsign.wavi@colin","custom_catch-up_and_book_a_meeting.wavi@colin","custom_github.wavi@colin","custom_jeeping.wavi@colin","custom_my_first_professional_computer.wavi@colin","custom_proverb.wavi@colin","custom_quote_of_the_day.wavi@colin","discord.wavi@colin","email.wavi@colin","facebook.wavi@colin","field_order_of_self.wavi@colin","firstname.wavi@colin","followers_of_self.at_follows.wavi@colin","following_by_self.at_follows.wavi@colin","image.wavi@colin","instagram.wavi@colin","lastname.wavi@colin","linkedin.wavi@colin","location.wavi@colin","locationnickname.wavi@colin","phone.wavi@colin","privateaccount.wavi@colin","profilepic.buzz@colin","pronoun.wavi@colin","publickey","publickey@colin","signing_publickey@colin","theme.wavi@colin","theme_color.wavi@colin","twitter.wavi@colin","youtube.wavi@colin"]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:40:58.494342|InboundListener|RCVD: [_6308d506-e67e-4037-9b5f-d1f2ca6cda99] lookup:publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:58.494385|AtSecondaryServer|inside _executeVerbCallBack: lookup:publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:58.494442|DefaultVerbExecutor|verb handler found : LookupVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:58.494496|LookupVerbHandler|fromAtSign : null
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |  atSign : @colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |  key : publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:58.494503|LookupVerbHandler|lookup_key in lookupVerbHandler : public:publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:58.497076|Secondary_Util|result : MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAiE/FJPmcsByxwTSUWSTQcz/08YkI956dL10+3IT4jiEqlkBTs+/6bp4NiAK/MWBW3smJLbK+g02mXetG/URqH0kb6HgcD4Idb0wfYqjT5IGKCbPA/YsO0gINaaBLaMkyzHyg7C/jdQNiJR+1zrJeHt/EtU1R7hN4tdpW1ZagiyrmMUkNH5kqXNzM9K/C28zRctBhlpNpwbg41vJ4eMRipYepl0s5uUfAGOoLhRRbzbN+40iALcnL648U8XzwU3SWoEZHvA7b+Xwu1l13kopXyWyudOciZgQoSa5UYxbT8HNhy5ptTQE13UW1JyNXoDNhb/no0y2SliOtWmkZl4U55wIDAQAB
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:58.497392|LookupVerbHandler|Verb : lookup  Response: Response{_data: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAiE/FJPmcsByxwTSUWSTQcz/08YkI956dL10+3IT4jiEqlkBTs+/6bp4NiAK/MWBW3smJLbK+g02mXetG/URqH0kb6HgcD4Idb0wfYqjT5IGKCbPA/YsO0gINaaBLaMkyzHyg7C/jdQNiJR+1zrJeHt/EtU1R7hN4tdpW1ZagiyrmMUkNH5kqXNzM9K/C28zRctBhlpNpwbg41vJ4eMRipYepl0s5uUfAGOoLhRRbzbN+40iALcnL648U8XzwU3SWoEZHvA7b+Xwu1l13kopXyWyudOciZgQoSa5UYxbT8HNhy5ptTQE13UW1JyNXoDNhb/no0y2SliOtWmkZl4U55wIDAQAB, _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:40:58.497434|InboundConnectionImpl|SENT: [_6308d506-e67e-4037-9b5f-d1f2ca6cda99] data:MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAiE/FJPmcsByxwTSUWSTQcz/08YkI956dL10+3IT4jiEqlkBTs+/6bp4NiAK/MWBW3smJLbK+g02mXetG/URqH0kb6HgcD4Idb0wfYqjT5IGKCbPA/YsO0gINaaBLaMkyzHyg7C/jdQNiJR+1zrJeHt/EtU1R7hN4tdpW1ZagiyrmMUkNH5kqXNzM9K/C28zRctBhlpNpwbg41vJ4eMRipYepl0s5uUfAGOoLhRRbzbN+40iALcnL648U8XzwU3SWoEZHvA7b+Xwu1l13kopXyWyudOciZgQoSa5UYxbT8HNhy5ptTQE13UW1JyNXoDNhb/no0y2SliOtWmkZl4U55wIDAQAB
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:58.584915|InboundConnectionImpl|InternetAddress('10.10.0.15', IPv4):11161 Disconnected
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | WARNING|2022-06-08 19:40:59.504387|AtSecondaryServer|ServerSocket.listen called onError with 'HandshakeException: Connection terminated during handshake'
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:59.610445|AtSecondaryServer|In _listen - clientSocket.peerCertificate : null
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:40:59.610673|InboundConnectionImpl|SENT: [_fe7fc1fc-a848-4eb0-a3c3-b643fa556b35] @
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:40:59.704042|InboundListener|RCVD: [_fe7fc1fc-a848-4eb0-a3c3-b643fa556b35] scan
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:59.704110|AtSecondaryServer|inside _executeVerbCallBack: scan
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:59.704175|DefaultVerbExecutor|verb handler found : ScanVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:59.711969|ScanVerbHandler|response.data : [custom_about_me.wavi@colin, custom_butterflies.wavi@colin, custom_callsign.wavi@colin, custom_catch-up_and_book_a_meeting.wavi@colin, custom_github.wavi@colin, custom_jeeping.wavi@colin, custom_my_first_professional_computer.wavi@colin, custom_proverb.wavi@colin, custom_quote_of_the_day.wavi@colin, discord.wavi@colin, email.wavi@colin, facebook.wavi@colin, field_order_of_self.wavi@colin, firstname.wavi@colin, followers_of_self.at_follows.wavi@colin, following_by_self.at_follows.wavi@colin, image.wavi@colin, instagram.wavi@colin, lastname.wavi@colin, linkedin.wavi@colin, location.wavi@colin, locationnickname.wavi@colin, phone.wavi@colin, privateaccount.wavi@colin, profilepic.buzz@colin, pronoun.wavi@colin, publickey, publickey@colin, signing_publickey@colin, theme.wavi@colin, theme_color.wavi@colin, twitter.wavi@colin, youtube.wavi@colin]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:59.712048|ScanVerbHandler|keysArray : [custom_about_me.wavi@colin, custom_butterflies.wavi@colin, custom_callsign.wavi@colin, custom_catch-up_and_book_a_meeting.wavi@colin, custom_github.wavi@colin, custom_jeeping.wavi@colin, custom_my_first_professional_computer.wavi@colin, custom_proverb.wavi@colin, custom_quote_of_the_day.wavi@colin, discord.wavi@colin, email.wavi@colin, facebook.wavi@colin, field_order_of_self.wavi@colin, firstname.wavi@colin, followers_of_self.at_follows.wavi@colin, following_by_self.at_follows.wavi@colin, image.wavi@colin, instagram.wavi@colin, lastname.wavi@colin, linkedin.wavi@colin, location.wavi@colin, locationnickname.wavi@colin, phone.wavi@colin, privateaccount.wavi@colin, profilepic.buzz@colin, pronoun.wavi@colin, publickey, publickey@colin, signing_publickey@colin, theme.wavi@colin, theme_color.wavi@colin, twitter.wavi@colin, youtube.wavi@colin], 33
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:59.712124|ScanVerbHandler|Verb : scan  Response: Response{_data: ["custom_about_me.wavi@colin","custom_butterflies.wavi@colin","custom_callsign.wavi@colin","custom_catch-up_and_book_a_meeting.wavi@colin","custom_github.wavi@colin","custom_jeeping.wavi@colin","custom_my_first_professional_computer.wavi@colin","custom_proverb.wavi@colin","custom_quote_of_the_day.wavi@colin","discord.wavi@colin","email.wavi@colin","facebook.wavi@colin","field_order_of_self.wavi@colin","firstname.wavi@colin","followers_of_self.at_follows.wavi@colin","following_by_self.at_follows.wavi@colin","image.wavi@colin","instagram.wavi@colin","lastname.wavi@colin","linkedin.wavi@colin","location.wavi@colin","locationnickname.wavi@colin","phone.wavi@colin","privateaccount.wavi@colin","profilepic.buzz@colin","pronoun.wavi@colin","publickey","publickey@colin","signing_publickey@colin","theme.wavi@colin","theme_color.wavi@colin","twitter.wavi@colin","youtube.wavi@colin"], _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:40:59.712158|InboundConnectionImpl|SENT: [_fe7fc1fc-a848-4eb0-a3c3-b643fa556b35] data:["custom_about_me.wavi@colin","custom_butterflies.wavi@colin","custom_callsign.wavi@colin","custom_catch-up_and_book_a_meeting.wavi@colin","custom_github.wavi@colin","custom_jeeping.wavi@colin","custom_my_first_professional_computer.wavi@colin","custom_proverb.wavi@colin","custom_quote_of_the_day.wavi@colin","discord.wavi@colin","email.wavi@colin","facebook.wavi@colin","field_order_of_self.wavi@colin","firstname.wavi@colin","followers_of_self.at_follows.wavi@colin","following_by_self.at_follows.wavi@colin","image.wavi@colin","instagram.wavi@colin","lastname.wavi@colin","linkedin.wavi@colin","location.wavi@colin","locationnickname.wavi@colin","phone.wavi@colin","privateaccount.wavi@colin","profilepic.buzz@colin","pronoun.wavi@colin","publickey","publickey@colin","signing_publickey@colin","theme.wavi@colin","theme_color.wavi@colin","twitter.wavi@colin","youtube.wavi@colin"]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:40:59.938370|InboundListener|RCVD: [_fe7fc1fc-a848-4eb0-a3c3-b643fa556b35] lookup:publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:59.938472|AtSecondaryServer|inside _executeVerbCallBack: lookup:publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:59.938643|DefaultVerbExecutor|verb handler found : LookupVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:59.938822|LookupVerbHandler|fromAtSign : null
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |  atSign : @colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |  key : publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:59.938860|LookupVerbHandler|lookup_key in lookupVerbHandler : public:publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:59.939277|Secondary_Util|result : MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAiE/FJPmcsByxwTSUWSTQcz/08YkI956dL10+3IT4jiEqlkBTs+/6bp4NiAK/MWBW3smJLbK+g02mXetG/URqH0kb6HgcD4Idb0wfYqjT5IGKCbPA/YsO0gINaaBLaMkyzHyg7C/jdQNiJR+1zrJeHt/EtU1R7hN4tdpW1ZagiyrmMUkNH5kqXNzM9K/C28zRctBhlpNpwbg41vJ4eMRipYepl0s5uUfAGOoLhRRbzbN+40iALcnL648U8XzwU3SWoEZHvA7b+Xwu1l13kopXyWyudOciZgQoSa5UYxbT8HNhy5ptTQE13UW1JyNXoDNhb/no0y2SliOtWmkZl4U55wIDAQAB
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:40:59.939774|LookupVerbHandler|Verb : lookup  Response: Response{_data: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAiE/FJPmcsByxwTSUWSTQcz/08YkI956dL10+3IT4jiEqlkBTs+/6bp4NiAK/MWBW3smJLbK+g02mXetG/URqH0kb6HgcD4Idb0wfYqjT5IGKCbPA/YsO0gINaaBLaMkyzHyg7C/jdQNiJR+1zrJeHt/EtU1R7hN4tdpW1ZagiyrmMUkNH5kqXNzM9K/C28zRctBhlpNpwbg41vJ4eMRipYepl0s5uUfAGOoLhRRbzbN+40iALcnL648U8XzwU3SWoEZHvA7b+Xwu1l13kopXyWyudOciZgQoSa5UYxbT8HNhy5ptTQE13UW1JyNXoDNhb/no0y2SliOtWmkZl4U55wIDAQAB, _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:40:59.939813|InboundConnectionImpl|SENT: [_fe7fc1fc-a848-4eb0-a3c3-b643fa556b35] data:MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAiE/FJPmcsByxwTSUWSTQcz/08YkI956dL10+3IT4jiEqlkBTs+/6bp4NiAK/MWBW3smJLbK+g02mXetG/URqH0kb6HgcD4Idb0wfYqjT5IGKCbPA/YsO0gINaaBLaMkyzHyg7C/jdQNiJR+1zrJeHt/EtU1R7hN4tdpW1ZagiyrmMUkNH5kqXNzM9K/C28zRctBhlpNpwbg41vJ4eMRipYepl0s5uUfAGOoLhRRbzbN+40iALcnL648U8XzwU3SWoEZHvA7b+Xwu1l13kopXyWyudOciZgQoSa5UYxbT8HNhy5ptTQE13UW1JyNXoDNhb/no0y2SliOtWmkZl4U55wIDAQAB
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:00.043688|InboundConnectionImpl|InternetAddress('10.10.0.15', IPv4):16874 Disconnected
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:07.375174|InboundConnectionImpl|SENT: [_f90d655f-375b-4617-8653-a8eb360537f9] notification: {"id":"-1","from":"@colin","to":"@colin","key":"statsNotification.@colin","value":"353754","operation":"update","epochMillis":1654717267375}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | WARNING|2022-06-08 19:41:08.341556|AtSecondaryServer|ServerSocket.listen called onError with 'HandshakeException: Connection terminated during handshake'
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:08.442435|AtSecondaryServer|In _listen - clientSocket.peerCertificate : null
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:08.442555|InboundConnectionImpl|SENT: [_f02fb15c-bac0-43f0-9d91-26a138219107] @
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:08.530109|InboundListener|RCVD: [_f02fb15c-bac0-43f0-9d91-26a138219107] scan
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:08.530156|AtSecondaryServer|inside _executeVerbCallBack: scan
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:08.530201|DefaultVerbExecutor|verb handler found : ScanVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:08.538758|ScanVerbHandler|response.data : [custom_about_me.wavi@colin, custom_butterflies.wavi@colin, custom_callsign.wavi@colin, custom_catch-up_and_book_a_meeting.wavi@colin, custom_github.wavi@colin, custom_jeeping.wavi@colin, custom_my_first_professional_computer.wavi@colin, custom_proverb.wavi@colin, custom_quote_of_the_day.wavi@colin, discord.wavi@colin, email.wavi@colin, facebook.wavi@colin, field_order_of_self.wavi@colin, firstname.wavi@colin, followers_of_self.at_follows.wavi@colin, following_by_self.at_follows.wavi@colin, image.wavi@colin, instagram.wavi@colin, lastname.wavi@colin, linkedin.wavi@colin, location.wavi@colin, locationnickname.wavi@colin, phone.wavi@colin, privateaccount.wavi@colin, profilepic.buzz@colin, pronoun.wavi@colin, publickey, publickey@colin, signing_publickey@colin, theme.wavi@colin, theme_color.wavi@colin, twitter.wavi@colin, youtube.wavi@colin]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:08.538837|ScanVerbHandler|keysArray : [custom_about_me.wavi@colin, custom_butterflies.wavi@colin, custom_callsign.wavi@colin, custom_catch-up_and_book_a_meeting.wavi@colin, custom_github.wavi@colin, custom_jeeping.wavi@colin, custom_my_first_professional_computer.wavi@colin, custom_proverb.wavi@colin, custom_quote_of_the_day.wavi@colin, discord.wavi@colin, email.wavi@colin, facebook.wavi@colin, field_order_of_self.wavi@colin, firstname.wavi@colin, followers_of_self.at_follows.wavi@colin, following_by_self.at_follows.wavi@colin, image.wavi@colin, instagram.wavi@colin, lastname.wavi@colin, linkedin.wavi@colin, location.wavi@colin, locationnickname.wavi@colin, phone.wavi@colin, privateaccount.wavi@colin, profilepic.buzz@colin, pronoun.wavi@colin, publickey, publickey@colin, signing_publickey@colin, theme.wavi@colin, theme_color.wavi@colin, twitter.wavi@colin, youtube.wavi@colin], 33
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:08.538900|ScanVerbHandler|Verb : scan  Response: Response{_data: ["custom_about_me.wavi@colin","custom_butterflies.wavi@colin","custom_callsign.wavi@colin","custom_catch-up_and_book_a_meeting.wavi@colin","custom_github.wavi@colin","custom_jeeping.wavi@colin","custom_my_first_professional_computer.wavi@colin","custom_proverb.wavi@colin","custom_quote_of_the_day.wavi@colin","discord.wavi@colin","email.wavi@colin","facebook.wavi@colin","field_order_of_self.wavi@colin","firstname.wavi@colin","followers_of_self.at_follows.wavi@colin","following_by_self.at_follows.wavi@colin","image.wavi@colin","instagram.wavi@colin","lastname.wavi@colin","linkedin.wavi@colin","location.wavi@colin","locationnickname.wavi@colin","phone.wavi@colin","privateaccount.wavi@colin","profilepic.buzz@colin","pronoun.wavi@colin","publickey","publickey@colin","signing_publickey@colin","theme.wavi@colin","theme_color.wavi@colin","twitter.wavi@colin","youtube.wavi@colin"], _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:08.538924|InboundConnectionImpl|SENT: [_f02fb15c-bac0-43f0-9d91-26a138219107] data:["custom_about_me.wavi@colin","custom_butterflies.wavi@colin","custom_callsign.wavi@colin","custom_catch-up_and_book_a_meeting.wavi@colin","custom_github.wavi@colin","custom_jeeping.wavi@colin","custom_my_first_professional_computer.wavi@colin","custom_proverb.wavi@colin","custom_quote_of_the_day.wavi@colin","discord.wavi@colin","email.wavi@colin","facebook.wavi@colin","field_order_of_self.wavi@colin","firstname.wavi@colin","followers_of_self.at_follows.wavi@colin","following_by_self.at_follows.wavi@colin","image.wavi@colin","instagram.wavi@colin","lastname.wavi@colin","linkedin.wavi@colin","location.wavi@colin","locationnickname.wavi@colin","phone.wavi@colin","privateaccount.wavi@colin","profilepic.buzz@colin","pronoun.wavi@colin","publickey","publickey@colin","signing_publickey@colin","theme.wavi@colin","theme_color.wavi@colin","twitter.wavi@colin","youtube.wavi@colin"]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:08.761394|InboundListener|RCVD: [_f02fb15c-bac0-43f0-9d91-26a138219107] lookup:publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:08.761456|AtSecondaryServer|inside _executeVerbCallBack: lookup:publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:08.761513|DefaultVerbExecutor|verb handler found : LookupVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:08.761573|LookupVerbHandler|fromAtSign : null
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |  atSign : @colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |  key : publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:08.761581|LookupVerbHandler|lookup_key in lookupVerbHandler : public:publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:08.762581|Secondary_Util|result : MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAiE/FJPmcsByxwTSUWSTQcz/08YkI956dL10+3IT4jiEqlkBTs+/6bp4NiAK/MWBW3smJLbK+g02mXetG/URqH0kb6HgcD4Idb0wfYqjT5IGKCbPA/YsO0gINaaBLaMkyzHyg7C/jdQNiJR+1zrJeHt/EtU1R7hN4tdpW1ZagiyrmMUkNH5kqXNzM9K/C28zRctBhlpNpwbg41vJ4eMRipYepl0s5uUfAGOoLhRRbzbN+40iALcnL648U8XzwU3SWoEZHvA7b+Xwu1l13kopXyWyudOciZgQoSa5UYxbT8HNhy5ptTQE13UW1JyNXoDNhb/no0y2SliOtWmkZl4U55wIDAQAB
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:08.763527|LookupVerbHandler|Verb : lookup  Response: Response{_data: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAiE/FJPmcsByxwTSUWSTQcz/08YkI956dL10+3IT4jiEqlkBTs+/6bp4NiAK/MWBW3smJLbK+g02mXetG/URqH0kb6HgcD4Idb0wfYqjT5IGKCbPA/YsO0gINaaBLaMkyzHyg7C/jdQNiJR+1zrJeHt/EtU1R7hN4tdpW1ZagiyrmMUkNH5kqXNzM9K/C28zRctBhlpNpwbg41vJ4eMRipYepl0s5uUfAGOoLhRRbzbN+40iALcnL648U8XzwU3SWoEZHvA7b+Xwu1l13kopXyWyudOciZgQoSa5UYxbT8HNhy5ptTQE13UW1JyNXoDNhb/no0y2SliOtWmkZl4U55wIDAQAB, _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:08.763572|InboundConnectionImpl|SENT: [_f02fb15c-bac0-43f0-9d91-26a138219107] data:MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAiE/FJPmcsByxwTSUWSTQcz/08YkI956dL10+3IT4jiEqlkBTs+/6bp4NiAK/MWBW3smJLbK+g02mXetG/URqH0kb6HgcD4Idb0wfYqjT5IGKCbPA/YsO0gINaaBLaMkyzHyg7C/jdQNiJR+1zrJeHt/EtU1R7hN4tdpW1ZagiyrmMUkNH5kqXNzM9K/C28zRctBhlpNpwbg41vJ4eMRipYepl0s5uUfAGOoLhRRbzbN+40iALcnL648U8XzwU3SWoEZHvA7b+Xwu1l13kopXyWyudOciZgQoSa5UYxbT8HNhy5ptTQE13UW1JyNXoDNhb/no0y2SliOtWmkZl4U55wIDAQAB
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:08.861769|InboundConnectionImpl|InternetAddress('10.10.0.15', IPv4):3950 Disconnected
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:09.926763|AtSecondaryServer|In _listen - clientSocket.peerCertificate : null
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:09.926874|InboundConnectionImpl|SENT: [_9d8ff8a8-7769-461c-8940-b9dcd955c22d] @
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:10.013390|InboundListener|RCVD: [_9d8ff8a8-7769-461c-8940-b9dcd955c22d] scan
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:10.013445|AtSecondaryServer|inside _executeVerbCallBack: scan
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:10.013502|DefaultVerbExecutor|verb handler found : ScanVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:10.021495|ScanVerbHandler|response.data : [custom_about_me.wavi@colin, custom_butterflies.wavi@colin, custom_callsign.wavi@colin, custom_catch-up_and_book_a_meeting.wavi@colin, custom_github.wavi@colin, custom_jeeping.wavi@colin, custom_my_first_professional_computer.wavi@colin, custom_proverb.wavi@colin, custom_quote_of_the_day.wavi@colin, discord.wavi@colin, email.wavi@colin, facebook.wavi@colin, field_order_of_self.wavi@colin, firstname.wavi@colin, followers_of_self.at_follows.wavi@colin, following_by_self.at_follows.wavi@colin, image.wavi@colin, instagram.wavi@colin, lastname.wavi@colin, linkedin.wavi@colin, location.wavi@colin, locationnickname.wavi@colin, phone.wavi@colin, privateaccount.wavi@colin, profilepic.buzz@colin, pronoun.wavi@colin, publickey, publickey@colin, signing_publickey@colin, theme.wavi@colin, theme_color.wavi@colin, twitter.wavi@colin, youtube.wavi@colin]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:10.021580|ScanVerbHandler|keysArray : [custom_about_me.wavi@colin, custom_butterflies.wavi@colin, custom_callsign.wavi@colin, custom_catch-up_and_book_a_meeting.wavi@colin, custom_github.wavi@colin, custom_jeeping.wavi@colin, custom_my_first_professional_computer.wavi@colin, custom_proverb.wavi@colin, custom_quote_of_the_day.wavi@colin, discord.wavi@colin, email.wavi@colin, facebook.wavi@colin, field_order_of_self.wavi@colin, firstname.wavi@colin, followers_of_self.at_follows.wavi@colin, following_by_self.at_follows.wavi@colin, image.wavi@colin, instagram.wavi@colin, lastname.wavi@colin, linkedin.wavi@colin, location.wavi@colin, locationnickname.wavi@colin, phone.wavi@colin, privateaccount.wavi@colin, profilepic.buzz@colin, pronoun.wavi@colin, publickey, publickey@colin, signing_publickey@colin, theme.wavi@colin, theme_color.wavi@colin, twitter.wavi@colin, youtube.wavi@colin], 33
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:10.021658|ScanVerbHandler|Verb : scan  Response: Response{_data: ["custom_about_me.wavi@colin","custom_butterflies.wavi@colin","custom_callsign.wavi@colin","custom_catch-up_and_book_a_meeting.wavi@colin","custom_github.wavi@colin","custom_jeeping.wavi@colin","custom_my_first_professional_computer.wavi@colin","custom_proverb.wavi@colin","custom_quote_of_the_day.wavi@colin","discord.wavi@colin","email.wavi@colin","facebook.wavi@colin","field_order_of_self.wavi@colin","firstname.wavi@colin","followers_of_self.at_follows.wavi@colin","following_by_self.at_follows.wavi@colin","image.wavi@colin","instagram.wavi@colin","lastname.wavi@colin","linkedin.wavi@colin","location.wavi@colin","locationnickname.wavi@colin","phone.wavi@colin","privateaccount.wavi@colin","profilepic.buzz@colin","pronoun.wavi@colin","publickey","publickey@colin","signing_publickey@colin","theme.wavi@colin","theme_color.wavi@colin","twitter.wavi@colin","youtube.wavi@colin"], _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:10.021683|InboundConnectionImpl|SENT: [_9d8ff8a8-7769-461c-8940-b9dcd955c22d] data:["custom_about_me.wavi@colin","custom_butterflies.wavi@colin","custom_callsign.wavi@colin","custom_catch-up_and_book_a_meeting.wavi@colin","custom_github.wavi@colin","custom_jeeping.wavi@colin","custom_my_first_professional_computer.wavi@colin","custom_proverb.wavi@colin","custom_quote_of_the_day.wavi@colin","discord.wavi@colin","email.wavi@colin","facebook.wavi@colin","field_order_of_self.wavi@colin","firstname.wavi@colin","followers_of_self.at_follows.wavi@colin","following_by_self.at_follows.wavi@colin","image.wavi@colin","instagram.wavi@colin","lastname.wavi@colin","linkedin.wavi@colin","location.wavi@colin","locationnickname.wavi@colin","phone.wavi@colin","privateaccount.wavi@colin","profilepic.buzz@colin","pronoun.wavi@colin","publickey","publickey@colin","signing_publickey@colin","theme.wavi@colin","theme_color.wavi@colin","twitter.wavi@colin","youtube.wavi@colin"]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:10.247440|InboundListener|RCVD: [_9d8ff8a8-7769-461c-8940-b9dcd955c22d] lookup:publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:10.247536|AtSecondaryServer|inside _executeVerbCallBack: lookup:publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:10.247583|DefaultVerbExecutor|verb handler found : LookupVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:10.247670|LookupVerbHandler|fromAtSign : null
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |  atSign : @colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |  key : publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:10.247680|LookupVerbHandler|lookup_key in lookupVerbHandler : public:publickey@colin
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:10.248286|Secondary_Util|result : MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAiE/FJPmcsByxwTSUWSTQcz/08YkI956dL10+3IT4jiEqlkBTs+/6bp4NiAK/MWBW3smJLbK+g02mXetG/URqH0kb6HgcD4Idb0wfYqjT5IGKCbPA/YsO0gINaaBLaMkyzHyg7C/jdQNiJR+1zrJeHt/EtU1R7hN4tdpW1ZagiyrmMUkNH5kqXNzM9K/C28zRctBhlpNpwbg41vJ4eMRipYepl0s5uUfAGOoLhRRbzbN+40iALcnL648U8XzwU3SWoEZHvA7b+Xwu1l13kopXyWyudOciZgQoSa5UYxbT8HNhy5ptTQE13UW1JyNXoDNhb/no0y2SliOtWmkZl4U55wIDAQAB
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:10.248568|LookupVerbHandler|Verb : lookup  Response: Response{_data: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAiE/FJPmcsByxwTSUWSTQcz/08YkI956dL10+3IT4jiEqlkBTs+/6bp4NiAK/MWBW3smJLbK+g02mXetG/URqH0kb6HgcD4Idb0wfYqjT5IGKCbPA/YsO0gINaaBLaMkyzHyg7C/jdQNiJR+1zrJeHt/EtU1R7hN4tdpW1ZagiyrmMUkNH5kqXNzM9K/C28zRctBhlpNpwbg41vJ4eMRipYepl0s5uUfAGOoLhRRbzbN+40iALcnL648U8XzwU3SWoEZHvA7b+Xwu1l13kopXyWyudOciZgQoSa5UYxbT8HNhy5ptTQE13UW1JyNXoDNhb/no0y2SliOtWmkZl4U55wIDAQAB, _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:10.248628|InboundConnectionImpl|SENT: [_9d8ff8a8-7769-461c-8940-b9dcd955c22d] data:MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAiE/FJPmcsByxwTSUWSTQcz/08YkI956dL10+3IT4jiEqlkBTs+/6bp4NiAK/MWBW3smJLbK+g02mXetG/URqH0kb6HgcD4Idb0wfYqjT5IGKCbPA/YsO0gINaaBLaMkyzHyg7C/jdQNiJR+1zrJeHt/EtU1R7hN4tdpW1ZagiyrmMUkNH5kqXNzM9K/C28zRctBhlpNpwbg41vJ4eMRipYepl0s5uUfAGOoLhRRbzbN+40iALcnL648U8XzwU3SWoEZHvA7b+Xwu1l13kopXyWyudOciZgQoSa5UYxbT8HNhy5ptTQE13UW1JyNXoDNhb/no0y2SliOtWmkZl4U55wIDAQAB
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:10.323379|InboundConnectionImpl|InternetAddress('10.10.0.15', IPv4):13392 Disconnected
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:12.362530|InboundListener|RCVD: [_977307f7-8a5d-463b-a4e4-613af49e8656] noop:0
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:12.362595|AtSecondaryServer|inside _executeVerbCallBack: noop:0
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:12.362735|DefaultVerbExecutor|verb handler found : NoOpVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:12.362923|NoOpVerbHandler|Verb : noop  Response: Response{_data: ok, _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:12.362977|InboundConnectionImpl|SENT: [_977307f7-8a5d-463b-a4e4-613af49e8656] data:ok
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:21.612786|InboundListener|RCVD: [_f90d655f-375b-4617-8653-a8eb360537f9] noop:0
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:21.612834|AtSecondaryServer|inside _executeVerbCallBack: noop:0
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:21.612894|DefaultVerbExecutor|verb handler found : NoOpVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-08 19:41:21.612985|NoOpVerbHandler|Verb : noop  Response: Response{_data: ok, _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-08 19:41:21.613007|InboundConnectionImpl|SENT: [_f90d655f-375b-4617-8653-a8eb360537f9] data:ok
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
^C
cconstab@canary-01:~$

cconstab commented 2 years ago

If I run from Trunk I can login (it uses the keys that were aledged to be faulty) and then get stuck at Sync in progress with lots of nasty errors in the logs..

e.g.

ried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-08 12:49:04.789382|AtLookup|AtLookup.findSecondary received atsign: jenmonroe
flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@jenmonroe"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-08 12:49:04.802383|AtLookup|AtLookup.findSecondary got answer: 97ef7af4-e033-59b5-bf63-bddd3d738eaa.swarm0001.atsign.zone:11030 and closing connection
flutter: FINER|2022-06-08 12:49:04.845382|AtLookup|AtLookup.findSecondary got answer: 446b7a28-8028-558d-be62-b079bf387bc0.canary.atsign.zone:1101 and closing connection
flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@jenmonroe"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-08 12:49:04.906384|AtLookup|AtLookup.findSecondary got answer: 446b7a28-8028-558d-be62-b079bf387bc0.canary.atsign.zone:1101 and closing connection
flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@jenmonroe"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.

cconstab commented 2 years ago

I have bumped this to P1 as I am concerned about this hitting other folks using desktop

@gkc FEEL FREE TO BUMP BACK DOWN IF YOU DO NOT AGREE

gkc commented 2 years ago

@cconstab I agree, those are ugly exceptions in the logs @sachins-geekyants have you diagnosed the problem? what help do you need?

sachins-geekyants commented 2 years ago

Initially, we assumed the sync issue on windows might be happening because of old dependency of at_client and at_client_mobile, I tried running @colin atsign on windows using atmopshere trunk and it failed for me too even after keeping the app running for around 20 mins.

Sync kind of freezed with Exception

flutter: SEVERE|2022-06-09 13:54:06.098748|AtLookup|Exception in sending to server, Exception: No response after 10000 millis from remote secondary

flutter: SEVERE|2022-06-09 13:54:06.099694|AtLookup|Error in remote verb execution Exception: No response after 10000 millis from remote secondary

flutter: SEVERE|2022-06-09 13:54:06.100691|SyncService|Exception in sync 84a62690-47bf-4c02-b796-5db3051fa883. Reason AT0023: Exception: No response after 10000 millis from remote secondary

flutter: SEVERE|2022-06-09 13:54:14.646045|AtLookup|Exception in sending to server, ErrorCode: AT0014 - Exception: Unexpected response found 

flutter: SEVERE|2022-06-09 13:54:14.646045|AtLookup|Error in remote verb execution ErrorCode: AT0014 - Exception: Unexpected response found

flutter: SEVERE|2022-06-09 13:54:14.647099|SyncUtil|Exception occurred in processing stats verb AT0014 - ErrorCode: AT0014 - Exception: Unexpected response found

flutter: INFO|2022-06-09 13:54:14.658010|SyncService|Returning the serverCommitId 357293

flutter: SEVERE|2022-06-09 13:54:15.092906|SyncUtil|Exception occurred in processing stats verb AT0003 - Invalid syntax : Invalid syntax. stats:1,2,3

I don't think app has control over sync. Hada call with @kalluriramkumar also regarding this issue.

@colin_logs.txt

sachins-geekyants commented 2 years ago

Apart from the sync Exception, noticed bug in the front end.

Regarding periodic refresh of history screen. Currently, we start periodic refreshes after onboarding completes. Which should be started after sync completes.

Exeption in getSentFileItems

flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@jenmonroe"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-08 12:49:04.845382|AtLookup|AtLookup.findSecondary got answer: 446b7a28-8028-558d-be62-b079bf387bc0.canary.atsign.zone:1101 and closing connection
flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@jenmonroe"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@jenmonroe"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.

This is happening when we are not able to convert any sent file JSON string to the view model, the JSON string data might be in old format and might be missing some required fields.

But these two exceptions should not cause any issue in sync process.

sachins-geekyants commented 2 years ago

Regarding not being able to authenticate even after providing correct key, we have an open issue for it. It was observed with @barbara atsing. #585.

We don't know the reason behind it, Nitesh looked into it but could not reproduce it.

cconstab commented 2 years ago

I can reproduce every time

sachins-geekyants commented 2 years ago

@cconstab can you think of any user behaviour that might have caused to fail auth? like some combination using multiple apps/atsign or atsign being deleted and onboarded again ?

cconstab commented 2 years ago

I have tried other @signs on my desktop and same result.. Very odd

sachins-geekyants commented 2 years ago

Very odd that it did not worked with other atsign aslo.

Can you check the at_client and at_client_mobile version mentioned in the pubspec.yaml and pubspec.lock file ?
Can you try running release tag version with other atsign and still see auth issue ?

If you are running from terminal, please use trunk branch.

cconstab commented 2 years ago

Using trunk and Flutter version

Flutter 3.0.2 • channel stable • https://github.com/flutter/flutter.git Framework • revision cd41fdd495 (2 days ago) • 2022-06-08 09:52:13 -0700 Engine • revision f15f824b57 Tools • Dart 2.17.3 • DevTools 2.12.2

PS C:\Users\colin\GitHub\@foundation\atmosphere_pro> flutter pub upgrade Resolving dependencies... _fe_analyzer_shared 40.0.0 analyzer 4.1.0 archive 3.3.0 args 2.3.1 asn1lib 1.1.0 async 2.8.2 (2.9.0 available)

at_backupkey_flutter 4.0.2 (was 4.0.1) at_base2e15 1.0.0 at_client 3.0.25 at_client_mobile 3.1.16 ! at_common_flutter 2.0.8 from git https://github.com/atsign-foundation/at_widgets at 265555 in at_common_flutter (overridden) at_commons 3.0.16 at_contact 3.0.5 ! at_contacts_flutter 4.0.5 from git https://github.com/atsign-foundation/at_widgets at 265555 in at_contacts_flutter (overridden) ! at_contacts_group_flutter 4.0.5 from git https://github.com/atsign-foundation/at_widgets at 265555 in at_contacts_group_flutter (overridden) at_lookup 3.0.24 ! at_onboarding_flutter 4.0.3 from git https://github.com/atsign-foundation/at_widgets at 265555 in at_onboarding_flutter (overridden) at_persistence_secondary_server 3.0.25 (was 3.0.24) at_persistence_spec 2.0.5 at_server_status 1.0.3 at_sync_ui_flutter 1.0.3 (was 1.0.2) at_utf7 1.0.0 at_utils 3.0.9 biometric_storage 3.0.1 (4.1.2 available) boolean_selector 2.1.0 carousel_slider 4.1.1 characters 1.2.0 (1.2.1 available) charcode 1.3.1 cli_dialog 0.5.0 cli_util 0.3.5 clock 1.1.0 collection 1.16.0 convert 3.0.2 (was 3.0.1) coverage 1.3.2 (was 1.0.3) cron 0.3.2 (0.5.0 available) cross_file 0.3.3+1 (was 0.3.3) crypto 3.0.2 crypton 2.0.5 (was 2.0.3) csslib 0.17.2 (was 0.17.1) cupertino_icons 0.1.3 (1.0.5 available) dart_console 1.0.0 dbus 0.7.4 (was 0.7.3) (0.7.5 available) desktop_window 0.4.0 emoji_picker_flutter 1.2.1 (was 1.1.2) encrypt 5.0.1 fake_async 1.3.0 ffi 1.2.1 (2.0.0 available) file 6.1.2 file_picker 4.6.1 (was 4.5.1) ! file_saver 0.1.0 from git https://github.com/incrediblezayed/file_saver.git at c0ea84 (overridden) file_selector 0.8.4+3 (was 0.8.4+2) file_selector_linux 0.0.2+1 file_selector_macos 0.8.2+2 (was 0.8.2+1) file_selector_platform_interface 2.0.4 file_selector_web 0.8.1+5 file_selector_windows 0.8.2+2 (was 0.8.2+1) filesystem_picker 2.0.1 (was 2.0.0) fixnum 1.0.1 flutter 0.0.0 from sdk flutter flutter_image_compress 1.1.0 flutter_keychain 2.1.0 flutter_local_notifications 9.6.0 (was 9.5.3+1) flutter_local_notifications_linux 0.5.0+1 (was 0.4.2) flutter_local_notifications_platform_interface 5.0.0 flutter_plugin_android_lifecycle 2.0.6 flutter_qr_reader 1.0.5 flutter_slidable 0.6.0 (1.3.0 available) flutter_test 0.0.0 from sdk flutter flutter_toastr 1.0.3 flutter_web_plugins 0.0.0 from sdk flutter fluttertoast 8.0.9 frontend_server_client 2.1.3 get_it 7.2.0 glob 2.0.2 google_fonts 2.3.3 (was 2.3.2) (3.0.1 available) hive 2.2.2 (was 2.1.0) html 0.15.0 http 0.13.4 http_multi_server 3.2.0 http_parser 4.0.1 (was 4.0.0) image 3.2.0 (was 3.1.3) internet_connection_checker 0.0.1+4 (was 0.0.1+3) intl 0.17.0 io 1.0.3 js 0.6.4 lints 1.0.1 (2.0.0 available) local_notifier 0.1.5 (was 0.1.4) logging 1.0.2 matcher 0.12.11 material_color_utilities 0.1.4 (0.1.5 available) meta 1.7.0 (1.8.0 available) mime 1.0.2 mocktail 0.3.0 msix 3.6.2 mutex 3.0.0 nested 1.0.0 new_version 0.3.1 (was 0.3.0) node_preamble 2.0.1 open_file 3.2.1 package_config 2.0.2 package_info_plus 1.4.2 package_info_plus_linux 1.0.5 package_info_plus_macos 1.3.0 package_info_plus_platform_interface 1.0.2 package_info_plus_web 1.0.5 package_info_plus_windows 1.0.5 path 1.8.1 (1.8.2 available) path_provider 2.0.11 (was 2.0.10) path_provider_android 2.0.14 path_provider_ios 2.0.9 path_provider_linux 2.1.7 (was 2.1.6) path_provider_macos 2.0.6 path_provider_platform_interface 2.0.4 path_provider_windows 2.0.7 (was 2.0.6) (2.1.0 available) pedantic 1.11.1 (discontinued replaced by lints) permission_handler 9.2.0 permission_handler_android 9.0.2+1 permission_handler_apple 9.0.4 permission_handler_platform_interface 3.7.0 permission_handler_windows 0.1.0 petitparser 5.0.0 (was 4.4.0) pin_code_fields 7.4.0 platform 3.1.0 plugin_platform_interface 2.1.2 pointycastle 3.6.0 pool 1.5.0 process 4.2.4 provider 6.0.3 (was 6.0.2) pub_semver 2.1.1 qr 2.1.0 (3.0.1 available) qr_flutter 4.0.0 receive_sharing_intent 1.4.5 rxdart 0.27.4 (was 0.27.3) share_plus 4.0.7 (was 4.0.4) share_plus_linux 3.0.0 share_plus_macos 3.0.1 (was 3.0.0) share_plus_platform_interface 3.0.3 (was 3.0.2) share_plus_web 3.0.1 (was 3.0.0) share_plus_windows 3.0.1 (was 3.0.0) shared_preferences 2.0.15 shared_preferences_android 2.0.12 shared_preferences_ios 2.1.1 shared_preferences_linux 2.1.1 shared_preferences_macos 2.0.4 shared_preferences_platform_interface 2.0.0 shared_preferences_web 2.0.4 shared_preferences_windows 2.1.1 shelf 1.3.0 shelf_packages_handler 3.0.0 shelf_static 1.1.0 shelf_web_socket 1.0.1 sky_engine 0.0.99 from sdk flutter source_map_stack_trace 2.1.0 source_maps 0.10.10 source_span 1.8.2 (1.9.0 available) stack_trace 1.10.0 stream_channel 2.1.0 string_scanner 1.1.0 (1.1.1 available) term_glyph 1.2.0 test 1.21.1 test_api 0.4.9 test_core 0.4.13 timezone 0.8.0 typed_data 1.3.1 (was 1.3.0) url_launcher 6.1.3 (was 6.1.2) url_launcher_android 6.0.17 url_launcher_ios 6.0.17 (was 6.0.16) url_launcher_linux 3.0.1 url_launcher_macos 3.0.1 url_launcher_platform_interface 2.0.5 url_launcher_web 2.0.11 url_launcher_windows 3.0.1 uuid 3.0.6 vector_math 2.1.2 video_compress 3.1.0 video_thumbnail 0.4.6 (0.5.0 available) vm_service 8.3.0 (was 7.5.0) watcher 1.0.1 web_socket_channel 2.2.0 webkit_inspection_protocol 1.1.0 (was 1.0.1) webview_flutter 3.0.4 webview_flutter_android 2.8.11 (was 2.8.8) webview_flutter_platform_interface 1.9.1 (was 1.8.1) webview_flutter_wkwebview 2.8.1 (was 2.7.5) win32 2.6.1 (was 2.5.2) (2.7.0 available) xdg_directories 0.2.0+1 xml 6.1.0 (was 5.3.1) yaml 3.1.1 zxing2 0.1.0 Warning: You are using these overridden dependencies: ! at_common_flutter 2.0.8 from git https://github.com/atsign-foundation/at_widgets at 265555 in at_common_flutter ! at_contacts_flutter 4.0.5 from git https://github.com/atsign-foundation/at_widgets at 265555 in at_contacts_flutter ! at_contacts_group_flutter 4.0.5 from git https://github.com/atsign-foundation/at_widgets at 265555 in at_contacts_group_flutter ! at_onboarding_flutter 4.0.3 from git https://github.com/atsign-foundation/at_widgets at 265555 in at_onboarding_flutter ! file_saver 0.1.0 from git https://github.com/incrediblezayed/file_saver.git at c0ea84 Changed 50 dependencies!

cconstab commented 2 years ago

App still hangs in the sync phase when using @ Colin

I see this in the client logs ....

flutter: FINER|2022-06-10 16:15:17.729350|SyncService|syncing to local: localCommitId 331775 serverCommitId 368353

flutter: FINER|2022-06-10 16:15:17.729350|SyncService|** syncBuilder sync:from:331775:limit:10:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})

flutter: FINER|2022-06-10 16:15:17.746926|SyncService|** syncBuilder sync:from:331775:limit:10:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})

flutter: SEVERE|2022-06-10 16:15:29.200970|AtLookup|Exception in sending to server, Exception: No response after 10000 millis from remote secondary 

flutter: SEVERE|2022-06-10 16:15:29.200970|AtLookup|Error in remote verb execution Exception: No response after 10000 millis from remote secondary

flutter: SEVERE|2022-06-10 16:15:29.200970|SyncService|Exception in sync 5445f1ce-84e4-49e0-920a-188de7402bc0. Reason AT0023: Exception: No response after 10000 millis from remote secondary

flutter: FINER|2022-06-10 16:15:32.236639|Monitor|received response on monitor: notification: {"id":"-1","from":"@colin","to":"@colin","key":"statsNotification.@colin","value":"368353","operation":"update","epochMillis":1654902932387}

flutter: FINER|2022-06-10 16:15:32.237638|SyncService|got stats notification in sync: 368353

and this in the secondary

79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:19.645313|InboundListener|RCVD: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:19.645510|AtSecondaryServer|inside _executeVerbCallBack: stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:19.645666|DefaultVerbExecutor|verb handler found : StatsVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:19.645770|StatsVerbHandler|In statsVerbHandler statID : :3, regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:19.645817|StatsVerbHandler|addStatToResult for id : 3, regex: (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:19.645903|RegexUtil|In commitID getMetrics...regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:19.665471|StatsVerbHandler|Verb : stats  Response: Response{_data: [{"id":"3","name":"lastCommitID","value":"368353"}], _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:19.665581|InboundConnectionImpl|SENT: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] data: [{"id":"3","name":"lastCommitID","value":"368353"}]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:26.950292|InboundConnectionImpl|SENT: [_5e5714a3-2654-4472-8d06-1440d5edff2b] notification: {"id":"-1","from":"@colin","to":"@colin","key":"statsNotification.@colin","value":"368353","operation":"update","epochMillis":1654902866950}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:31.071564|InboundListener|RCVD: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:31.071667|AtSecondaryServer|inside _executeVerbCallBack: stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:31.071731|DefaultVerbExecutor|verb handler found : StatsVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:31.071814|StatsVerbHandler|In statsVerbHandler statID : :3, regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:31.071847|StatsVerbHandler|addStatToResult for id : 3, regex: (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:31.071869|RegexUtil|In commitID getMetrics...regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:31.091578|StatsVerbHandler|Verb : stats  Response: Response{_data: [{"id":"3","name":"lastCommitID","value":"368353"}], _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:31.091696|InboundConnectionImpl|SENT: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] data: [{"id":"3","name":"lastCommitID","value":"368353"}]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:38.022413|InboundListener|RCVD: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:38.022465|AtSecondaryServer|inside _executeVerbCallBack: stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:38.022516|DefaultVerbExecutor|verb handler found : StatsVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:38.022563|StatsVerbHandler|In statsVerbHandler statID : :3, regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:38.022577|StatsVerbHandler|addStatToResult for id : 3, regex: (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:38.022588|RegexUtil|In commitID getMetrics...regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:38.040308|StatsVerbHandler|Verb : stats  Response: Response{_data: [{"id":"3","name":"lastCommitID","value":"368353"}], _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:38.040393|InboundConnectionImpl|SENT: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] data: [{"id":"3","name":"lastCommitID","value":"368353"}]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:47.384164|InboundConnectionImpl|SENT: [_5e5714a3-2654-4472-8d06-1440d5edff2b] notification: {"id":"-1","from":"@colin","to":"@colin","key":"statsNotification.@colin","value":"368353","operation":"update","epochMillis":1654902887384}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:49.474420|InboundListener|RCVD: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:49.474498|AtSecondaryServer|inside _executeVerbCallBack: stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:49.474580|DefaultVerbExecutor|verb handler found : StatsVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:49.474699|StatsVerbHandler|In statsVerbHandler statID : :3, regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:49.474777|StatsVerbHandler|addStatToResult for id : 3, regex: (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:49.474808|RegexUtil|In commitID getMetrics...regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:49.487068|StatsVerbHandler|Verb : stats  Response: Response{_data: [{"id":"3","name":"lastCommitID","value":"368353"}], _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:49.487182|InboundConnectionImpl|SENT: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] data: [{"id":"3","name":"lastCommitID","value":"368353"}]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:53.116573|InboundListener|RCVD: [_5e5714a3-2654-4472-8d06-1440d5edff2b] noop:0
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:53.116654|AtSecondaryServer|inside _executeVerbCallBack: noop:0
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:53.116709|DefaultVerbExecutor|verb handler found : NoOpVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:14:53.116855|NoOpVerbHandler|Verb : noop  Response: Response{_data: ok, _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:14:53.116875|InboundConnectionImpl|SENT: [_5e5714a3-2654-4472-8d06-1440d5edff2b] data:ok
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:00.945327|InboundListener|RCVD: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:00.945397|AtSecondaryServer|inside _executeVerbCallBack: stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:00.945466|DefaultVerbExecutor|verb handler found : StatsVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:00.945529|StatsVerbHandler|In statsVerbHandler statID : :3, regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:00.945559|StatsVerbHandler|addStatToResult for id : 3, regex: (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:00.945577|RegexUtil|In commitID getMetrics...regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:00.959765|StatsVerbHandler|Verb : stats  Response: Response{_data: [{"id":"3","name":"lastCommitID","value":"368353"}], _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:00.960036|InboundConnectionImpl|SENT: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] data: [{"id":"3","name":"lastCommitID","value":"368353"}]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:11.953148|InboundConnectionImpl|SENT: [_5e5714a3-2654-4472-8d06-1440d5edff2b] notification: {"id":"-1","from":"@colin","to":"@colin","key":"statsNotification.@colin","value":"368353","operation":"update","epochMillis":1654902911953}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:12.285575|InboundListener|RCVD: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:12.285651|AtSecondaryServer|inside _executeVerbCallBack: stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:12.285706|DefaultVerbExecutor|verb handler found : StatsVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:12.285752|StatsVerbHandler|In statsVerbHandler statID : :3, regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:12.285768|StatsVerbHandler|addStatToResult for id : 3, regex: (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:12.285776|RegexUtil|In commitID getMetrics...regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:12.299220|StatsVerbHandler|Verb : stats  Response: Response{_data: [{"id":"3","name":"lastCommitID","value":"368353"}], _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:12.299292|InboundConnectionImpl|SENT: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] data: [{"id":"3","name":"lastCommitID","value":"368353"}]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:15.333553|InboundListener|RCVD: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:15.333639|AtSecondaryServer|inside _executeVerbCallBack: stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:15.333795|DefaultVerbExecutor|verb handler found : StatsVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:15.333900|StatsVerbHandler|In statsVerbHandler statID : :3, regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:15.333922|StatsVerbHandler|addStatToResult for id : 3, regex: (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:15.333946|RegexUtil|In commitID getMetrics...regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:15.350150|StatsVerbHandler|Verb : stats  Response: Response{_data: [{"id":"3","name":"lastCommitID","value":"368353"}], _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:15.350294|InboundConnectionImpl|SENT: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] data: [{"id":"3","name":"lastCommitID","value":"368353"}]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:15.437540|InboundListener|RCVD: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:15.437650|AtSecondaryServer|inside _executeVerbCallBack: stats:3:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:15.437719|DefaultVerbExecutor|verb handler found : StatsVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:15.437779|StatsVerbHandler|In statsVerbHandler statID : :3, regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:15.437812|StatsVerbHandler|addStatToResult for id : 3, regex: (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:15.437839|RegexUtil|In commitID getMetrics...regex : (.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:15.456456|StatsVerbHandler|Verb : stats  Response: Response{_data: [{"id":"3","name":"lastCommitID","value":"368353"}], _type: null, _isError: false, _errorMessage: null}
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:15.456539|InboundConnectionImpl|SENT: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] data: [{"id":"3","name":"lastCommitID","value":"368353"}]
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:17.972958|InboundListener|RCVD: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] sync:from:331775:limit:10:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:17.973014|AtSecondaryServer|inside _executeVerbCallBack: sync:from:331775:limit:10:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:17.973091|DefaultVerbExecutor|verb handler found : SyncProgressiveVerbHandler
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | INFO|2022-06-10 23:15:18.146841|InboundListener|RCVD: [_e869c813-72f0-4d9d-8d5c-4ff46c35dc7e] sync:from:331775:limit:10:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:18.146900|AtSecondaryServer|inside _executeVerbCallBack: sync:from:331775:limit:10:(.mospherepro|atconnections|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    |
79b6d83f-5026-5fda-8299-5a0704bd2416_secondary.1.vqrbfbqc7uxa@canary-02.us-central1-b.c.secondaries.internal    | FINER|2022-06-10 23:15:18.146973|DefaultVerbExecutor|verb handler found : SyncProgressiveVerbHandler

cconstab commented 2 years ago

I am now blocked as both atmosphere and wavi will not sync and the atsign @ colin is active on them both so I cannot reset the atsign

gkc commented 2 years ago

Just to re-confirm - this is still a problem only on the desktop apps - ie not the Android or iOS apps?

cconstab commented 2 years ago

Some application clues

flutter: FINER|2022-06-10 16:51:04.537647|AtLookup|AtLookup.findSecondary got answer: null and closing connection 

[ERROR:flutter/lib/ui/ui_dart_state.cc(198)] Unhandled Exception: Exception: Unable to find secondary address for atSign:kevin
#0      CacheableSecondaryAddressFinder._updateCache (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:92:9)
<asynchronous suspension>
#1      CacheableSecondaryAddressFinder.findSecondary (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:47:7)
<asynchronous suspension>
#2      AtClientValidation.isAtSignExists (package:at_client/src/util/at_client_validation.dart:51:7)
<asynchronous suspension>

flutter: FINER|2022-06-10 16:51:04.544647|AtLookup|AtLookup.findSecondary got answer: null and closing connection

[ERROR:flutter/lib/ui/ui_dart_state.cc(198)] Unhandled Exception: Exception: Unable to find secondary address for atSign:kevin
#0      CacheableSecondaryAddressFinder._updateCache (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:92:9)
<asynchronous suspension>
#1      CacheableSecondaryAddressFinder.findSecondary (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:47:7)
<asynchronous suspension>
#2      AtClientValidation.isAtSignExists (package:at_client/src/util/at_client_validation.dart:51:7)
<asynchronous suspension>

flutter: FINER|2022-06-10 16:51:04.559645|AtLookup|AtLookup.findSecondary received atsign: kevin

flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@kevin"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-10 16:51:04.618683|AtLookup|AtLookup.findSecondary received atsign: kevin 

flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@kevin"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-10 16:51:04.626684|AtLookup|AtLookup.findSecondary got answer: null and closing connection

[ERROR:flutter/lib/ui/ui_dart_state.cc(198)] Unhandled Exception: Exception: Unable to find secondary address for atSign:kevin
#0      CacheableSecondaryAddressFinder._updateCache (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:92:9)
<asynchronous suspension>
#1      CacheableSecondaryAddressFinder.findSecondary (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:47:7)
<asynchronous suspension>
#2      AtClientValidation.isAtSignExists (package:at_client/src/util/at_client_validation.dart:51:7)
<asynchronous suspension>

flutter: FINER|2022-06-10 16:51:04.666684|AtLookup|AtLookup.findSecondary received atsign: kevin 

flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@kevin"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-10 16:51:04.673684|AtLookup|AtLookup.findSecondary got answer: null and closing connection

[ERROR:flutter/lib/ui/ui_dart_state.cc(198)] Unhandled Exception: Exception: Unable to find secondary address for atSign:kevin
#0      CacheableSecondaryAddressFinder._updateCache (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:92:9)
<asynchronous suspension>
#1      CacheableSecondaryAddressFinder.findSecondary (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:47:7)
<asynchronous suspension>
#2      AtClientValidation.isAtSignExists (package:at_client/src/util/at_client_validation.dart:51:7)
<asynchronous suspension>

flutter: FINER|2022-06-10 16:51:04.716679|AtLookup|AtLookup.findSecondary received atsign: kevin 

flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@kevin"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-10 16:51:04.725252|AtLookup|AtLookup.findSecondary got answer: null and closing connection

[ERROR:flutter/lib/ui/ui_dart_state.cc(198)] Unhandled Exception: Exception: Unable to find secondary address for atSign:kevin
#0      CacheableSecondaryAddressFinder._updateCache (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:92:9)
<asynchronous suspension>
#1      CacheableSecondaryAddressFinder.findSecondary (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:47:7)
<asynchronous suspension>
#2      AtClientValidation.isAtSignExists (package:at_client/src/util/at_client_validation.dart:51:7)
<asynchronous suspension>

flutter: FINER|2022-06-10 16:51:04.776682|AtLookup|AtLookup.findSecondary received atsign: kevin 

flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@kevin"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-10 16:51:04.835709|AtLookup|AtLookup.findSecondary got answer: null and closing connection 

[ERROR:flutter/lib/ui/ui_dart_state.cc(198)] Unhandled Exception: Exception: Unable to find secondary address for atSign:kevin
#0      CacheableSecondaryAddressFinder._updateCache (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:92:9)
<asynchronous suspension>
#1      CacheableSecondaryAddressFinder.findSecondary (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:47:7)
<asynchronous suspension>
#2      AtClientValidation.isAtSignExists (package:at_client/src/util/at_client_validation.dart:51:7)
<asynchronous suspension>

flutter: FINER|2022-06-10 16:51:04.835709|AtLookup|AtLookup.findSecondary got answer: null and closing connection

[ERROR:flutter/lib/ui/ui_dart_state.cc(198)] Unhandled Exception: Exception: Unable to find secondary address for atSign:kevin
#0      CacheableSecondaryAddressFinder._updateCache (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:92:9)
<asynchronous suspension>
#1      CacheableSecondaryAddressFinder.findSecondary (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:47:7)
<asynchronous suspension>
#2      AtClientValidation.isAtSignExists (package:at_client/src/util/at_client_validation.dart:51:7)
<asynchronous suspension>

flutter: FINER|2022-06-10 16:51:04.844248|AtLookup|AtLookup.findSecondary received atsign: kevin

flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@kevin"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-10 16:51:04.897620|AtLookup|AtLookup.findSecondary received atsign: kevin 

flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@kevin"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-10 16:51:04.942621|AtLookup|AtLookup.findSecondary got answer: null and closing connection 

[ERROR:flutter/lib/ui/ui_dart_state.cc(198)] Unhandled Exception: Exception: Unable to find secondary address for atSign:kevin
#0      CacheableSecondaryAddressFinder._updateCache (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:92:9)
<asynchronous suspension>
#1      CacheableSecondaryAddressFinder.findSecondary (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:47:7)
<asynchronous suspension>
#2      AtClientValidation.isAtSignExists (package:at_client/src/util/at_client_validation.dart:51:7)
<asynchronous suspension>

flutter: FINER|2022-06-10 16:51:04.944622|AtLookup|AtLookup.findSecondary received atsign: kevin

flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@kevin"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-10 16:51:04.972133|AtLookup|AtLookup.findSecondary got answer: null and closing connection 

[ERROR:flutter/lib/ui/ui_dart_state.cc(198)] Unhandled Exception: Exception: Unable to find secondary address for atSign:kevin
#0      CacheableSecondaryAddressFinder._updateCache (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:92:9)
<asynchronous suspension>
#1      CacheableSecondaryAddressFinder.findSecondary (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:47:7)
<asynchronous suspension>
#2      AtClientValidation.isAtSignExists (package:at_client/src/util/at_client_validation.dart:51:7)
<asynchronous suspension>

flutter: FINER|2022-06-10 16:51:04.995720|AtLookup|AtLookup.findSecondary received atsign: kevin 

flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@kevin"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-10 16:51:05.041719|AtLookup|AtLookup.findSecondary received atsign: kevin 

flutter: exeption in getSentFileItems : NoSuchMethodError: Class 'String' has no instance method 'forEach'.
Receiver: "@kevin"
Tried calling: forEach(Closure: (dynamic) => Null)
flutter: FINER|2022-06-10 16:51:05.049720|AtLookup|AtLookup.findSecondary got answer: null and closing connection

[ERROR:flutter/lib/ui/ui_dart_state.cc(198)] Unhandled Exception: Exception: Unable to find secondary address for atSign:kevin
#0      CacheableSecondaryAddressFinder._updateCache (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:92:9)
<asynchronous suspension>
#1      CacheableSecondaryAddressFinder.findSecondary (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:47:7)
<asynchronous suspension>
#2      AtClientValidation.isAtSignExists (package:at_client/src/util/at_client_validation.dart:51:7)
<asynchronous suspension>

flutter: FINER|2022-06-10 16:51:05.094758|AtLookup|AtLookup.findSecondary got answer: null and closing connection 

[ERROR:flutter/lib/ui/ui_dart_state.cc(198)] Unhandled Exception: Exception: Unable to find secondary address for atSign:kevin
#0      CacheableSecondaryAddressFinder._updateCache (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:92:9)
<asynchronous suspension>
#1      CacheableSecondaryAddressFinder.findSecondary (package:at_lookup/src/cache/cacheable_secondary_address_finder.dart:47:7)
<asynchronous suspension>
#2      AtClientValidation.isAtSignExists (package:at_client/src/util/at_client_validation.dart:51:7)
<asynchronous suspension>

cconstab commented 2 years ago

I have spent hours on this and see so many errors it is unclear to me what is going on

cconstab commented 2 years ago

Just to double check I deleted @ Colin on my Android phone and reinstalled and it worked just fine

sitaram-kalluri commented 2 years ago

@cconstab : I have tested with atmosphere pro with trunk branch code (onboarded the atsign for 3 to 4 times) on windows desktop version and unable to replicate the issue. The sync completes successfully. Attaching the video. Can you please try with atmosphere pro trunk branch code (if in case you are using other version)

https://user-images.githubusercontent.com/16645698/173192144-54b8c573-85f6-4a94-a12c-e1738575ecf7.mp4

Apologies, my laptop frozen and accidentally ticket was closed. Reopened the issue.

sitaram-kalluri commented 2 years ago

@cconstab @sachins-geekyants : What is the version of atmosphere pro used for testing (Is it different from what we have in trunk branch)?

cconstab commented 2 years ago

Interesting as I am using the trunk branch and it fails for me!

cconstab commented 2 years ago

As does wavi!

cconstab commented 2 years ago

OK after @kalluriramkumar observation I wondered if Starlink my new ISP had anything to do with this ?? It simply cannot right ???

Well I swapped to SierraTel and things just worked !

Now to figure out what on earth is going on !!

NETWORKS!

So I moved back to Starlink and guess what everythig still works OMG this is odd..

Eve odder @ wavi is also unstuck

cconstab commented 2 years ago

Ok I had the same synch problem on my MAC and so I wanted to figure out what was going on without changing ISP..

The solution ?

I chnaged the MTU on the WIFI interface to 1280 (normally set to 1500)

And then magically it worked !

Once working again I cannot get it to fail even when I move back to 1500..

BUT this is likely because we are not moving as much data and the frame is not full ??

This needs some more tests in a Dart program sending large chunks of data..

If anyone else hits this problem, this is the work around to get them up..

BUT this is super ugly !

This also explains why ANdroid was working as it likely has a lower MTU set....

My MTU is set correctly on the network... Here's some proof..

cconstab@cally:~$ ping google.com -s 1498 -M do
PING google.com (142.250.68.78) 1498(1526) bytes of data.
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
^C
--- google.com ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2025ms

cconstab@cally:~$

MTU discovery finds 1500 and so the stack should adjust accordingly...

We hit this same issue on the server/docker side but I think this is showing that either fragmentation or MTU discovery sis not working..

More investigation and test code to be written this week...

Also explains why nothing was obvious in the logs... Ho hum.. @kalluriramkumar this is all thanks to your video above THANK YOU !!!

Out of interest what is your MTU on your machine ?

sitaram-kalluri commented 2 years ago

@cconstab : MTU in my machine is 1500.

sitaram@sitaram-ThinkPad-E14:~$ ping google.com -s 1498 -M do
PING google.com(maa03s46-in-x0e.1e100.net (2404:6800:4007:82b::200e)) 1498 data bytes
ping: local error: message too long, mtu: 1500
ping: local error: message too long, mtu: 1500
ping: local error: message too long, mtu: 1500
^C
--- google.com ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2032ms

cconstab commented 2 years ago

I suspect the MTU on your router is lower..

sitaram-kalluri commented 2 years ago

I suspect the MTU on your router is lower..

Verified using ifconfig command and it shows mtu as 1500 (Please correct me if the command used is incorrect).

wlp0s20f3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.29.104 netmask 255.255.255.0 broadcast 192.168.29.255 inet6 2405:201:c00c:2136:e3d3:bc50:b790:e8cc prefixlen 64 scopeid 0x0

cconstab commented 2 years ago

Cool beans.. Mine is set up the same way.. Now I cannot get anything to fail, but all I did was change my MTU on my Mac.

Once working changed it back.

Something somewhere did not like 1500…

cconstab commented 2 years ago

Seems to be related to MTU discovery failing..

Many websites hit this issue and have turned down the MTU on the server to 1400 or so to reduce issues with MTU ..

Perhaps we consider the same which forces the TCP stack to advertise a MSS of 1400..

cconstab commented 1 year ago

This issue never found a 100% solution but also no longer is happening.

atsign-foundation / at_server

Sync in Progress on both @wavi and @mospherePro on Windows #738