Error: no matches for OriginalId certmanager.k8s.io_v1alpha1_ClusterIssuer|~X|letsencrypt-staging

[Kouruu]

After spending the weekend fighting with the original Arcus code with error after error, I settled down and decided to use this script. After all, I merely just need to get the Iris functionality back which I can then hopefully incorporate into home assistant. However running the setup-local.sh, I get this error and it fails. I'm sure I can figure it out over time but am tired. Wondering if anyone else has seen this error and knows how to get past it.

Error: no matches for OriginalId certmanager.k8s.io_v1alpha1_ClusterIssuer|~X|letsencrypt-staging; no matches for CurrentId certmanager.k8s.io_v1alpha1_ClusterIssuer|~X|letsencrypt-staging; failed to find unique target for patch certmanager.k8s.io_v1alpha1_ClusterIssuer|letsencrypt-staging error: no objects passed to apply

[AndrewX192]

I'm not familiar with that error, but I'm guessing that version 0.7 of cert-manager no longer works, and we will probably need to upgrade to a newer version. You could try changing the file that gets included as part of setup-local.sh

What functionality in Arcus are you hoping to incorporate into Home Assistant? Right now there's not a good way to expose functionality from Arcus to other platforms (but that's something I've been working on).

[Kouruu]

Tried using 0.8.1 but it still does it. I know it throws an error right before it

Error from server (AlreadyExists): secrets "shared" already exists
namespace/default labeled

Don't know if that has anything to do with it. As of right now, I have a Iris Thermostat and Alarm keypads, I'd like to get operational. I was good coding C in my day, but this is a learning curve I can tackle. If I could just get it installed.

Edit: Trashed the VM and started over hoping it would help. Alas, gives me the same exact error. :(

[AndrewX192]

That error isn't a problem - basically the setup script tries to create a secret based on the contents of "secret/" each time you run it - although kubernetes doesn't support/use secrets that way, and instead treats them as immutable (the opposite of configmaps, yay). If you make changes to your secrets you'll need to delete and recreate them, but otherwise it's fine to ignore this error.

[Kouruu]

function apply {
  ./kustomize build overlays/local-production-local/ | $KUBECTL apply -f -
}
retry 10 apply

Mind if I can get clarification on what this is doing? It's where the error is originating from.

[AndrewX192]

That block of code runs kustomize (a tool to template/override kubernetes templates) and applies the configuration. If it fails, it will attempt up to 10 times (see the definition of retry in the scripts). If you're having a problem, it's not likely with this block of code, but rather in the configuration itself (e.g. in config/ or overlays/)

[AndrewX192]

Also, it looks like there's a new problem with istio in microk8s due to microk8s being updated recently, see https://github.com/ubuntu/microk8s/issues/545

[Kouruu]

Not sure, I've tried installing microk8s using what was listed there, still same problem. I have noticed a problem with Istio. Kinda aggrevating really. In order to restart the installation, I have to uninstall microk8s to get it to not error out when it gets to Istio. microk8s.reset just hangs. I'll take another look at config. the overlays just populates the answers to the questions the script asks me. Unless I'm putting invalid info in it.

[AndrewX192]

Yeah, it seems that istio/microk8s is generally broken for new installations. I thought the edge version of microk8s would work, but I guess not - I wonder if it's possible to install an older version of the snap that doesn't have these problems?

Also yes - microk8s.reset not working is something I've ran into as well. If you have specific errors you're seeing when applying the overlay, I'd be happy to take a look at those. Otherwise, I don't have a good solution to the overall kubernetes ecosystem maturity issues I believe we're running into other than I might consider dumping kubernetes overall in favor of k3s or even just docker compose.

[Kouruu]

The specific error I was getting was trying to override my admin email on the place holder. To save myself some headache, I decided to just hard code it and remove the overlay for that part. Got me a little farther, but not much.

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "clusterissuers.admission.certmanager.k8s.io": the server is currently unable to handle the request

Found this installed so not sure.

customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io created

Going to try and use the default microk8s again and see what happens

[AndrewX192]

I believe I've run into that error a few times - it takes a while for cert-manager to become available so you just have to kind of wait for it to play out. The script will attempt to apply the configuration multiple times until cert-manager is ready.

[sgrayban]

I have been watching this repo for a while... Is it usable/stable yet or still trying to get it to work correctly ?

[Kouruu]

Or count to 10 and time out :-P

If that's the case, I can try to increase the number of times to see if it's needing the extra time.

[AndrewX192]

@sgrayban well, I've been using it for the last few months to run two Arcus environments, with several people on them, but I'm not aware of anyone other than me being able to get it running, which is unfortunate. Perhaps we should setup up some support channel or easier means of communication and work it out? There are some new people working on arcusplatform/arcusweb now, I might see if they can take some time and provide feedback, since we can iterate quicker that way.

[Kouruu]

Try #12, it got past it. Now gotta get past this hurdle.

Error from server (NotFound): pods "cassandra-0" not found

Progress. Making progress.

Edit:

Try number 10, it got past that one and finished the script. I'll pick up tomorrow evening and continue. Pending everything is running.

kouruu@arcus:~/arcus-k8$ microk8s.kubectl get pods
NAME                                    READY   STATUS     RESTARTS   AGE
alarm-service-769b768b9b-gdrmw          2/2     Running    3          10m
cassandra-0                             2/2     Running    0          10m
client-bridge-8658fc87fd-j8w2h          2/2     Running    3          10m
driver-services-56fc56c7b6-d9ndc        2/2     Running    3          10m
history-service-bb56d4844-l2xdr         2/2     Running    3          10m
hub-bridge-86789665c4-fpp6l             0/2     Init:0/1   0          10m
kafka-0                                 2/2     Running    4          10m
modelmanager-history-77pgs              1/2     Running    5          15m
modelmanager-platform-4m7ph             1/2     Running    5          15m
modelmanager-video-lwhp7                1/2     Running    5          15m
notification-services-b4bcd9b59-774ld   0/2     Init:0/1   0          10m
platform-services-77f7d6598d-fwpdp      2/2     Running    3          10m
rule-service-6849b6f55d-bvnfw           2/2     Running    3          10m
scheduler-service-68488bd499-d5d27      2/2     Running    4          10m
subsystem-service-65c5576966-hnwkc      2/2     Running    2          10m
ui-server-55f99d45fc-wkqg5              2/2     Running    0          10m
zookeeper-5ff79bfcd5-4bsp2              2/2     Running    0          10m

[sgrayban]

@AndrewX192 what are you running it on ?

[AndrewX192]

@Kouruu looks good! Notification service can't start unless you have APNS credentials + paid apple developed account (sorry about that), and hub-bridge won't start until you run ./setup-hubkeystore.sh. Make sure you can get into the Arcus UI with a valid production certificate before you setup the hub trust store though.. otherwise the Hub won't be able to connect.

EDIT: Also thanks to a Kubernetes/Istio limitation, once the modelmanager jobs finish (e.g. 1/2 is running) you should delete them - unfortunatelly a job doesn't finish until it's sidecar finishes and istio doesn't exit on it's own. There's a known issue in istio for this.

@sgrayban I'm running all my Arcus instances using the scripts provided in this repo (without any substantial modifications) inside of Ubuntu 18.04 virtual machines.

[sgrayban]

@AndrewX192 Does it really take all of this to run ? 12GB or more of RAM, and 15GB of disk space

[Kouruu]

It is what it is. If I don't need it for what I'm working on, I'm not really worried about it. Just trying to unbrick at least the useful Iris devices in the house and integrate them into a new setup. I'm also wondering, my initial issue that I basically bypassed. If I'd have increased the retry count to start with, would the initial problem disappeared with it. Perhaps the Cert Manager wasn't running initially when I was getting the merge errors. something to probably look into later. I might create another test machine and try it to see.

[AndrewX192]

@sgrayban my production instance is using 9GB of ram and a good 25GB of disk space. The drivers service in particular is very heavy on memory usage (almost 1GB) and kubernetes in general is not light on resources - although replacing it with k3s may help. You may be able to get it to run with less resources with some additional time invested, but I stopped at 8GB. Disk space usage depends on how busy your system is, and whether you decide to add system monitoring (the timeseries database will create tens of gigabytes in metrics).

@Kouruu it's entirely possible that having it run longer could have fixed it - I can imagine it may take longer for cert-manager to start on some systems. BTW if you're looking to port devices over, I'd recommend just looking at the device drivers in platform/arcus-containers/driver-services/src/main/resources and port them to your system of choice. The device model is very similar to that of SmartThings and Hubitat, so it shouldn't be very difficult to port - I've been using that to my advantage to add support for a bunch of new sensors in Arcus.

[sgrayban]

@AndrewX192 so it sounds like not really cost effective for personal use....

[AndrewX192]

@sgrayban Arcus isn't for everyone, especially today. If you want something that runs on minimal resources, then you have plenty of other options, like OpenHab or HomeAssistant (along with their caveats). If you want something that supports multiple homes, has a driver model similar to SmartThings and Hubitat, has a reasonable robust UI and apps for control, multi-user support, and a security system that still works offline, and a bunch of spare computing resources then Arcus might be the right option.

I'd also mention that computing resources continue to get very cheap - one could buy an older laptop, perhaps a ThinkPad to run Arcus for not a terribly large amount of money.

[sgrayban]

Already use Hubitat.... I'm just looking to get away from proprietor FW code and actually do what I want it do.

[AndrewX192]

@sgrayban You mean the code on the hub? Arcus struggles with the same problem at the moment, since the code for the zigbee and zwave controllers (e.g. the part that puts zigbee and zwave messages onto the serial bus) is not open source. Thankfully there's some good open source efforts (mainly openhab and zsmartsystems) which i think can be leveraged to get to an 80% solution.

I'm also thinking about adding MQTT support to Arcus so that it can be integrated with home assistant. Anyway, this is going pretty far off the topic of this original issue, so either start a thread on livingwithiris or send me an email if you want to discuss further.

[Kouruu]

Hey Andrew, is it normal to have a 404 not found when I access the UI?

A_-yD64': Get http://hub.arcussmarthome.com/.well-known/acme-challenge/s--9UE8J1W1Oz2V5vrSVRs0fLMZzl0ATltBXA_-yD64: dial tcp: lookup hub.arcussmarthome.com on 10.152.183.10:53: no such host

Also getting this error.

[AndrewX192]

It appears that you didn't set ARCUS_DOMAIN_NAME or fill it out in the setup script, so the ui-service setup a virtualhost of hub.arcussmarthome.com which isn't in public DNS.

Arcus requires a public-facing domain name, as it is HTTPs only and uses LetsEncrypt to automatically issue certificates - if the domain name is not resolvable, or the IP address and port it points to not reachable, then it will not be able to setup a certificate.

Since you've had issues getting setup-local to run, try

ARCUS_ADMIN_EMAIL='user@example.com' ARCUS_DOMAIN_NAME='arcus.example.com' ARCUS_SUBNET="172.16.6.1-172.16.6.4" bash ./script/shared-config.sh (with appropriate values entered in) and then microk8s.kubectl apply -f overlays/local-production-local/

[Kouruu]

I see what your saying, but before I go much further, you telling me I need a valid domain name to attach to my server for the arcus service to resolve? If so, any suggestions. I used to pay for a domain name but not anymore.

Got a domain. It was a bugger to get past 2 port forwarding schemas between CrappTT and pFsense. I'm getting errors trying to run the apply because error validating "overlays/local-production-local/kustomization.yaml": error validating data: [apiVersion not set, kind not set]; if you choose to ignore these errors, turn validation off with --validate=false I just changed the domain info manually. How'd you get past the login/registration page? Might be something stupid simple. However I've already seen what I've been looking at, what the guide says should happen and what I've experienced are 2 different things. EG I can't SSH into my hub due to access denied with right key on usb stick. But Eh. 1 thing at a time.

[Kouruu]

Got a new one for you:

 sudo ./setup-hubkeystore.sh
Creating hub-keystore...
unable to load key
139932716863936:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: ANY PRIVATE KEY

I've checked, the key files are empty and I've tried adding debug lines in the script to list the contents of the files as the script is running but is empty as well. Kinda frustrating. Can't even get to what I'm trying to do because I can't get past the setup process due to the errors I've encountered.

[AndrewX192]

You're seeing that error because the certificate didn't get issued - are you able to login to the Arcus UI without having to click through a certificate warning?

There's two other scripts I probably forgot to include in the setup instructions:

./useprodcert.sh (tells the system to use letsencrypt prod instead of staging, once you are confident in your network/DNS) ./setup-hubtruststore.sh (sets up the hub server tls key/cert based on the prod cert - it's in a different format and thus has to be converted periodically).

also, I understand your frustration in getting this up. I spend 20+ hours getting Arcus to work when I first started out, and it was not as functional or reliable as what these scripts offer. Unfortunately, the original marathon configuration was not open sourced, so there was a lot that had to be re-considered/learned in order to adapt this to something that someone else could get working. I would very much appreciate your help on documenting or closing any gaps you find, as right now there a lot of people struggling but I haven't been able to commit the same level of effort into this aspect of the project recently.

[Kouruu]

Well I can't persay login to the UI. However I do get to the website without the certificate warning. I haven't ran useprodcert.sh. I followed the instructions and edited the file to change staging to prod. I'll try today before I go to work. It's going to take me a little bit. I'm not that familiar with Yaml so I'm having to learn it. Nor am I a 100% on Linux. I didn't start playing with it until a few years ago. Got a better understanding of moving my way around the CLI. Then on top of it there are many new packages and different approaches to running services. I'm more than willing to help out getting this running. Just gotta play catchup. Probably wouldn't hurt if I can get some time to pour through the source code myself. And see what's doing what. All the errors are the frustrating thing. However once I can understand where they are coming from and what causes them. It makes things a little easier to identify them.

Hang in there. Once I can get this going, I'll help where I can.

EDIT:

Both were causing errors. But I figured out the problem.

KUBECTL=${KUBECTL:-kubectl}

My setup is Local on my network and does not acknowledge this. I changed to KUBECTL=/snap/bin/microk8s.kubectl and they ran properly. Even spat out the keys to console when I was trying to troubleshoot it.

I need to pick your brain from here though. So now by all means it should be up and running. However I have no way of logging into the web ui at the moment nor can I SSH into my hub. Did you have any issues here?

Last thing I got, I see I get more interaction out of the web ui. However the setup created an account with my email to which I don't have the password to and sending an email to reset is failing. :-/

[AndrewX192]

Well I can't persay login to the UI. However I do get to the website without the certificate warning.

So you get to a login page? You need to create an account. Unfortunately the version that is currently up has a bug and you'll have to fiddle around with reloading the page to get past that step. There's a change that should be landing in the next few days to fix this.

I haven't ran useprodcert.sh I don't understand how you wouldn't be getting a certificate warning then. Until you run useprodcert.sh you will be using the staging cert, which your browser won't trust.

It's going to take me a little bit ... It makes things a little easier to identify them. That's totally fine.

Both were causing errors. But I figured out the problem.

Ah yes! I thought I had documented that - when you use microk8s there's no kubectl command, bur rather there's microk8s.kubectl. So you need to be able to vary the command to call depending on where you're using k8s.

I need to pick your brain from here though. So now by all means it should be up and running. However I have no way of logging into the web ui at the moment nor can I SSH into my hub. Did you have any issues here?

I recommend looking at https://github.com/arcus-smart-home/arcusplatform/tree/master/tools/hubdebug and https://github.com/arcus-smart-home/arcusplatform/blob/master/docs/hub.md. I have a pre-packaged jar which you can use to replace the hub trust store, but regrettably it's not publicly accessible. Let me see about uploading that.

If you run into any specific issues, I'll try to follow up on these. For the UI I'd recommend looking at your brower's developer tools since that will show any websocket issues.

[Kouruu]

This is what I mean about not being able to SSH into my Hub using the password with my debug key on a USB stick plugged into the Hub.

kouruu@arcus:~/arcus-k8/secret$ ssh root@192.168.10.137
root@192.168.10.137's password:
Permission denied, please try again.
root@192.168.10.137's password:
Permission denied, please try again.
root@192.168.10.137's password:
root@192.168.10.137: Permission denied (publickey,password).
kouruu@arcus:~/arcus-k8/secret$ kz58!~Eb.RZ?+bqb

Of course trying with this password as stated in the MD file.

The second doc won't do me any good unless I can get past the first step. I've confirmed it's a V2 Hub but sadly the key they gave doesn't work with my hub.

Another thing I noticed, Create Account doesn't work on the webui login either.

[AndrewX192]

I've confirmed it's a V2 Hub but sadly the key they gave doesn't work with my hub.

I have a dozen V2 hubs and haven't seen that issue before, except when I was trying to login to the hub. Can you provide your hub id (e.g. XXX-12345)? The password for the v3 hub was intially wrong, but we were able to get this sorted out thanks to the former Iris team! Please remember that the v2 hub is a centralite IH200 (e.g. a white square box) and the third generation is an Great Star IH300 (a circular white device). There are other models, but I doubt you have one.

Another thing I noticed, Create Account doesn't work on the webui login either.

Can you clarify what exactly you're seeing? There is a known issue where it asks you to setup pro monitoring and you have to reload a few times - it's a bit tricky to explain the work around, but we're waiting for the fix to that to land. Can you provide screenshots (or preferably, specific technical details beyond what you've described)?

[Kouruu]

the hub is a IH200

LWG-1759

The website may have to wait. I tried to rerun the installation now that I have a domain name setup with my system and trying to get my production certificate to revalidate. Kinda broke it at the moment and Google Chrome thinks its an attacker.

NET::ERR_CERT_AUTHORITY_INVALID
Subject: Kubernetes Ingress Controller Fake Certificate

Issuer: Kubernetes Ingress Controller Fake Certificate

Expires on: Jul 27, 2020

Current date: Jul 28, 2019

BTW, I'm glad you've had so much luck with this. I don't think I've ever seen so many road blocks trying to set something up. It has literally fought me every step of the way.

[AndrewX192]

the hub is a IH200

Hm, I'm not sure what to think of your hub - the SSH server doesn't start until the key is recognized, and the password isn't configurable. Are you sure that you're logging into the right box? What happens if you disconnect it and try to SSH in? I think some more basic troubleshooting is in order.

Subject: Kubernetes Ingress Controller Fake Certificate

Yes, your browser has cause for alarm - don't accept this certificate. You're seeing this because cert-manager was not able to get a valid staging/production certificate. You'll need to make sure your system is reachable externally, and that cert-manager is working (look at kubectl log to understand what's going on).

I'm sorry you're having trouble getting this up - it's been quite an experience for me to get Arcus working, and I hope to get these issues worked out and documented or fixed shortly. While the software and concepts in Arcus are fairly mature, the infrastructure to host Arcus is relatively immature since it's being recreated in the open source version on different technology (Kubernetes instead of Apache Marathon).

[Kouruu]

Well I had it once but when I reran the script, it basically invalidated the first script. I got that part working again. I'm fixing to put a config file on my hub stick to see if I can link it to my setup to get it anywhere further. As to the answer to your question. my memory stick with the debug key allows it to attempt to SSH without it the connection refuses.

Well so I've tried resetting the Hub recopied the debug key and added a cfg and to no avail. I still cannot log into this God forsaken Hub. Right now after it boots it goes to a state of the green light flashing at a slow rate and the red light flashing at a medium rate. SSH as root still gives me a permission denied. Unplugging the Hub gives me a No route to host error. I'm at a loss right now.

[AndrewX192]

Regarding the hub, can you try a factory reset (hold the reset button for 30 seconds)? It sounds like something is wrong with your hub - since the key is being accepted, but you are unable to login.

[Kouruu]

Haven't had a chance to mess with it past few days. Busy with work. I have already reset it with the method you suggested. Didn't work. I'm just wondering if it's possible that maybe an update had to be released prior to the Iris shutdown that enabled the passwords for the hubs. (Speculating here.)

Although I know for certain that password definitely does not work with this hub.

[AndrewX192]

No, I don't think it's a version issue. Let me reach out to some folks and see if they can help. I have a dozen hubs, some of which are running really old firmware, and that password still works. I believe you can put the hub firmware on a USB stick and have it update from that file automatically, but in practice I've never seen that work.

You could connect to the hub over serial console and see if there's any messages there (you'd have to get a USB UART and connect it to J1 (see http://forum.livingwithiris.com/topic/4161-hacking-the-20-hub/).

If this doesn't work, it may be easier to just get another hub.

[Kouruu]

Ok....

So I bought another Hub. Still giving me a permission denied when I try to SSH into it using the password. Same as before, I've put the debug key on the USB that matches the hub. But it does the same exact thing as the other hub. Unless there is an unwritten step that I'm missing here, I now have 2 hubs that are utterly useless.

[AndrewX192]

All the steps are documented on https://github.com/wl-net/arcusplatform/blob/master/tools/hubdebug/hub_debug_keys%20_and_ssh_README.txt

I'd like to put something step by step (with pictures/screenshots), but thus far it hasn't really been an issue.

[Kouruu]

Andrew,

Up to this point I really appreciate all the help you have provided me to get this far. I could not have done this without you.

That said, I have done everything that page has said to do to SSH into my hubs. As can be seen in the screen shots, I have the dbg key for the hub I was working on and the other one my Putty terminal where I was logging in as root and using the password on that page for V2 hubs. What I know is I have tried this on 2 V2 Hubs, one I just bought. I also tried adding a configuration to it. I've tried on 2 different USB drives made sure both were formatted to FAT32. 1 drive is a 32Gb SanDisk. The other is a Transcend 2Gb which is fairly older. Wasn't sure if the Hub may not have supported the newer Dongle. Tried to rule everything out. I know having the dbg key for either Hub allows me the option to SSH in. Without which it gives me a Connection Refused. I just get stuck on the password part because every time I've tried the given password, and even possible combinations incase there was a typo, it gives me Access Denied. :-/

[AndrewX192]

I'm still at a loss for what's happening here - I'd like to minimize a few other variables just to be sure:

• Where and how are you copying the password? Some applications treat the "!" as a special character, so that could be a factor. I recommend going to the txt file, explicitly selecting the password and copying it.
• Terminal software - I have not tested putty, but believe it should work. If you want to replicate what I've been working off of, I recommend trying Fedora 30 with gnome-terminal.
• Hub and factory reset: since your hub does appear to be starting the SSH sever, I don't think this is a factor, but please hold down the reset button for 30 seconds to completely reset the hub. Once this is complete, you should get an SSH Host Key mismatch when connecting and you'll need to remedy that. If you do not see this warning, then it could mean that something else is wrong.

I also tested a few additional things locally, and I don't believe you need to include the cfg file or anything beyond what you've described. If you're curious, here's the SSH log when connecting: https://gist.github.com/AndrewX192/a61b636d06372acb15a85611bf10c8e4

UPDATE: I even got it to work in the latest version of Putty on Windows 10

[Kouruu]

• Where and how are you copying the password? Some applications treat the "!" as a special character, so that could be a factor. I recommend going to the txt file, explicitly selecting the password and copying it.

I've tried copy paste and typing it in. Even tried copying to text file and copied and typed from there into the password field. I've also typed the password onto the command line in putty to see what characters were popping out, just incase it was a different character set.

• Terminal software - I have not tested putty, but believe it should work. If you want to replicate what I've been working off of, I recommend trying Fedora 30 with gnome-terminal.

I've tried in Putty to directly connect to it and I've also had a putty session opened to my Arcus VM and tried to use ssh from there. See below.

• Hub and factory reset: since your hub does appear to be starting the SSH sever, I don't think this is a factor, but please hold down the reset button for 30 seconds to completely reset the hub. Once this is complete, you should get an SSH Host Key mismatch when connecting and you'll need to remedy that. If you do not see this warning, then it could mean that something else is wrong.

Tried this multiple times with both devices. In Putty, I just have to accept the new key. While using ssh on my Linux VM, I have to clear it out of the hosts file. Thanks for confirming that I'm not typing the password wrong, But I do admit, a bit frustrating for this to happen to me on 2 hubs. I did a debug ssh on my Linux VM like you had done if it could give any different clues as to whats going on. I'm suspecting the fact that I can't log into this hub is the main reason I can't get anywhere past the Login/Signup on the Web interface that I have forward facing.

kouruu@arcus:~$ ssh root@192.168.10.43 -vv
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "192.168.10.43" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 192.168.10.43 [192.168.10.43] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/kouruu/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/kouruu/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/kouruu/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/kouruu/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/kouruu/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/kouruu/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/kouruu/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/kouruu/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version dropbear_2014.63
debug1: no match: dropbear_2014.63
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 192.168.10.43:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,kexguess2@matt.ucc.asn.au
debug2: host key algorithms: ssh-rsa
debug2: ciphers ctos: aes128-ctr,3des-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes256-cbc,twofish256-cbc,twofish-cbc,twofish128-cbc
debug2: ciphers stoc: aes128-ctr,3des-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes256-cbc,twofish256-cbc,twofish-cbc,twofish128-cbc
debug2: MACs ctos: hmac-sha1-96,hmac-sha1,hmac-md5
debug2: MACs stoc: hmac-sha1-96,hmac-sha1,hmac-md5
debug2: compression ctos: zlib,zlib@openssh.com,none
debug2: compression stoc: zlib,zlib@openssh.com,none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:9fayb8JDgOn9dNUE1TC7IsEWZmQjhXIAY+DXzBKDw+s
The authenticity of host '192.168.10.43 (192.168.10.43)' can't be established.
RSA key fingerprint is SHA256:9fayb8JDgOn9dNUE1TC7IsEWZmQjhXIAY+DXzBKDw+s.
Are you sure you want to continue connecting (yes/no)? yes
Failed to add the host to the list of known hosts (/home/kouruu/.ssh/known_hosts).
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: /home/kouruu/.ssh/id_rsa ((nil))
debug2: key: /home/kouruu/.ssh/id_dsa ((nil))
debug2: key: /home/kouruu/.ssh/id_ecdsa ((nil))
debug2: key: /home/kouruu/.ssh/id_ed25519 ((nil))
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/kouruu/.ssh/id_rsa
debug1: Trying private key: /home/kouruu/.ssh/id_dsa
debug1: Trying private key: /home/kouruu/.ssh/id_ecdsa
debug1: Trying private key: /home/kouruu/.ssh/id_ed25519
debug2: we did not send a packet, disable method
debug1: Next authentication method: password
root@192.168.10.43's password:
debug2: we sent a password packet, wait for reply
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
root@192.168.10.43's password:
debug2: we sent a password packet, wait for reply
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
root@192.168.10.43's password:
debug2: we sent a password packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
root@192.168.10.43: Permission denied (publickey,password).
kouruu@arcus:~$ kz58!~Eb.RZ?+bqb

[AndrewX192]

I'm suspecting the fact that I can't log into this hub is the main reason I can't get anywhere past the Login/Signup on the Web interface that I have forward facing.

You should be able to create an account and get to the step where you add a hub (entering a hub id) without needing to have a hub setup. The two do not need to talk to each other until that point. If you still have trouble getting the platform/ui/login/registration to work, let me know.

Re the hub, I noticed that your hub reports that it is using "dropbear_2014.63", whereas mine typically report a 2016 version string, suggesting that you're using an older version of firmware. I only have a few hubs out of their boxes and ready to test against now, and many of them are hacked apart (e.g. extra wires attached) or otherwise not in a great place to test against, but I did verify that I have been able to get the firmware off of a hub using the 2014 version of dropbear. I also confirmed that the root password hash for this version of the firmware is the same as the current version.

There's one last thing we can try, and that is updating the firmware. Send me an email asking for it, and I'll get you a firmware image - then follow the instructions on https://github.com/wl-net/arcusplatform/blob/master/tools/hubdebug/hub_debug_keys%20_and_ssh_README.txt to attempt to update.

If this doesn't work, then I see a few different solutions:

1. I can expose a hub SSH server to you temporarily for the purpose of verifying that your password works.
2. I can mail you a known working hub in exchange for any non-working hubs (which I'll find a solution for)
3. You can get a USB / Serial interface cable and solder on a header to J1 and attempt to troubleshoot further yourself
4. You could also instruct the hub to connect back to you over HTTP using the cfg file and potentially update/compromise the system that way.

FYI, while I don't believe this is a critical error, I do see "Failed to add the host to the list of known hosts (/home/kouruu/.ssh/known_hosts)." in your error log.

[Kouruu]

This is what happens every time I try to create an account on the web service. I have tried every kind of browser (Chrome, Firefox, Edge, Safari, and IE). Obviously, I don't have an account setup. I wouldn't be mentioning it as an impasse if it was. I sent you an email in regards to the firmware. Only reason that one was so old because when I bought it, it was supposed to be new in the box. The other Hub I have may have a newer version on it as I got this while Iris was still active. Just haven't bothered checking it as they are both producing the same results.

[AndrewX192]

I've got four more hubs in box (mostly that shipped in ProMonitoring Starter Kits) that I'm going to try. I found out today that some really old hubs used a different password, but it's going to some time to locate that password. Short term, I would recommend trying to update the hub firmware with the image I sent you.

I just tested signup on my dev cluster (which is running HEAD version of arcus-k8 and containers) and was able to signup without an issue. Can you open your browser developer tools and try creating an account again and note any request/response from the /account/CreateAccount endpoint?

I suspect that the backend handling the request to create an account is unavailable / unable to talk to the database, and that the front-end is treating that as a generic "you already registered that email" condition. Please note that email addresses must be unique, but phone numbers do not have to be (which is actually very useful for testing)!

The logs for the client-bridge and platform-services deployments should tell you what is happening.

[sgrayban]

WOW you guys are already testing hubs ? Damn..... that's really cool. I can wait for a production write up !!

[AndrewX192]

I've been using Arcus since April 7th, shortly after the shut down, to some extent (it took about a month to get it to run reliably with push notifications and everything). I've been meaning to put together some writeups but I've not been able to make that a priority recently.

[sgrayban]

So you are reusing the old Iris hubs for this I gather ?

[AndrewX192]

Yes, the old hubs connect to the new platform with minimal modifications (it is necessary to replace the trust store with one that contains the certificate authority of LetsEncrypt, instead of the old IRIS root CA). I have a prototype of the IRIS firmware running on a Raspberry PI with dongles (although one of the radios doesn't work), and have also been working to rebuild the firmware and re-implement the zigbee/zwave stack on the hubs in order to make those fully open source as well.