Can't activate dess secondary on staging

[cpswan]

Describe the bug Blocker for #491

Unable to activate a @ cicd3 running on a dess VM:

In the app logs I see:

flutter: SEVERE|2022-02-21 15:10:31.063655|Onboarding Service|error in authenticating =>  Exception: Auth failed 
flutter: SEVERE|2022-02-21 15:10:31.063655|QR Scan|Error in authenticateWith cram secret

On the server I see:

cicd3_secondary.1.fmypp29n1sac@cicd1x64    | SEVERE|2022-02-21 15:10:27.233968|DefaultVerbExecutor|exception in processing command :cram:eb051b0393b1aa99da8167013380c126a3be2ff54b5fad4926377c5b652810d8ce5e9686c446bb740d9c1a1d2a42b9d2e43381c1d92cc046f52c8054c6f9d0fb: Exception: privatekey:at_secret does not exist in keystore
cicd3_secondary.1.fmypp29n1sac@cicd1x64    |
cicd3_secondary.1.fmypp29n1sac@cicd1x64    | WARNING|2022-02-21 15:10:27.234009|GlobalExceptionHandler|Exception: privatekey:at_secret does not exist in keystore

To Reproduce Steps to reproduce the behavior:

1. First I update .env on mwc_demo/flutter/iot_receiver to set ROOT_DOMAIN=root.atsign.wtf
2. Then I start the iot_receiver flutter run -d windows
3. And then click on SETUP NEW @SIGN
4. And then click on Upload QR code
5. Then I upload cicd3qr.PNG grabbed from sudo dess-reshowqr @cicd3 in an SSH session to the cicd1 VM

Expected behavior Atsign should be activated and .atKeys generated for download

[athandle]

@cpswan I checked entry was done in root, that's what I have control @murali-shris @kalluriramkumar @sarika01 can you please check this once and let me know if you need my help @tinashe404 can you please link this with QR code upload not working issue

[cconstab]

I have checked that the QR code contains the right CRAM key and that the secondary (cicd3) also has the right CRAM key..

I set up an app to point to root.atsign.wft and I see the same error in the log as Chris

cicd3_secondary.1.fmypp29n1sac@cicd1x64    | SEVERE|2022-02-21 16:08:01.303744|DefaultVerbExecutor|exception in processing command :cram:7db49ee48a5ad63267908648c0c1349a37e0c502e280b5d9a01dd813afbfbca824c1d4da8e71b7110e379b37fae248994da0d2383ccbc71ae4a6626075ae0e7d: Exception: privatekey:at_secret does not exist in keystore
cicd3_secondary.1.fmypp29n1sac@cicd1x64    |
cicd3_secondary.1.fmypp29n1sac@cicd1x64    | WARNING|2022-02-21 16:08:01.303775|GlobalExceptionHandler|Exception: privatekey:at_secret does not exist in keystore
cicd3_secondary.1.fmypp29n1sac@cicd1x64    |

To fault find further probably needs a secondary running dart interpreter so we can see more of what is going on..

[cconstab]

Looks to me that CRAM is not working on the atsigncompany/secondary:dess_cicd conatiner..

Using at_tools/at_cram I cannot authenticate even by hand

@from:@cicd3
data:_15c2c4dc-7faf-45e9-b2fa-7977c6fee4a2@cicd3:758aedb7-34cf-4af0-9b24-d720183ff27d
@cram:9c15818623b24f3dd3d94cc6833e41ac3eeadf96531d1e1246cdc1b806c07e3c229f2f8daf2e67b768b6db8b94abf8324f209e2bfe7a95d00e11cb622865ea7e
error:AT0015-key not found : privatekey:at_secret does not exist in keystore
@

Secondary was started correctly using

 -a @cicd3 -p 6465 -s fcc6fbbc66759dbc25237ff0438ecfab5******************************************************

[cconstab]

Migrated to prod secondary image and hit same issue..

Next idea to reset HIve boxes and start up secondary

Deleted hive boxes (forces clean CRAM) and success on the command line... Should now work in UI @cpswan to check

root@cicd1x64:/home/atsign/atsign/@cicd3# ls *
accessLog:
access_log_8c1391b95a6a327bb9418405387e648ba12d2dba40114a9a163daa65bffcfc2e.hive  access_log_8c1391b95a6a327bb9418405387e648ba12d2dba40114a9a163daa65bffcfc2e.lock

commitLog:
commit_log_8c1391b95a6a327bb9418405387e648ba12d2dba40114a9a163daa65bffcfc2e.hive  commit_log_8c1391b95a6a327bb9418405387e648ba12d2dba40114a9a163daa65bffcfc2e.lock

hive:
8c1391b95a6a327bb9418405387e648ba12d2dba40114a9a163daa65bffcfc2e.hash  8c1391b95a6a327bb9418405387e648ba12d2dba40114a9a163daa65bffcfc2e.hive  8c1391b95a6a327bb9418405387e648ba12d2dba40114a9a163daa65bffcfc2e.lock

notificationLog.v1:
notifications_8c1391b95a6a327bb9418405387e648ba12d2dba40114a9a163daa65bffcfc2e.hive  notifications_8c1391b95a6a327bb9418405387e648ba12d2dba40114a9a163daa65bffcfc2e.lock

storage:
root@cicd1x64:/home/atsign/atsign/@cicd3# rm accessLog/*
root@cicd1x64:/home/atsign/atsign/@cicd3# rm commitLog/*
root@cicd1x64:/home/atsign/atsign/@cicd3# rm hive/*
root@cicd1x64:/home/atsign/atsign/@cicd3# rm notificationLog.v1/*
root@cicd1x64:/home/atsign/atsign/@cicd3# ls *
accessLog:

commitLog:

hive:

notificationLog.v1:

storage:
root@cicd1x64:/home/atsign/atsign/@cicd3# docker service scale cicd3_secondary=1
cicd3_secondary scaled to 1
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged
root@cicd1x64:

then a openssl and check the cram works

root@cicd1x64:/home/atsign/atsign/@cicd3# openssl s_client 0:6465
@scan
data:["signing_publickey@cicd3"]
@from:@cicd3
data:_d5bfa322-b236-4cf5-a78a-7b648631184b@cicd3:745e2e2a-6337-48cd-a526-3b6429334130
@cram:f7ead311c262b1b01bf70fa684829dc5985f44cdc8ea3375ab551f76e1deff5f0e08ef58da526cba856ff3b82722dbedf1ef2aa7ade716e1da2e7bb80cac9c3a
data:success
@cicd3@
error:AT0003-Invalid syntax : invalid command
@cicd3@read:errno=0
root@cicd1x64:/home/atsign/atsign/@cicd3#

[cpswan]

Thanks @cconstab clearing down the state worked, though it's worrying that those secondaries were able to get into a state where CRAM wasn't working.