search

medic / chis-interoperability

5 stars 1 forks source link

Invalid username and password for root@openhim.org at OpenHIM Console #6

Closed nsikak-e-akpakpan closed 3 years ago

nsikak-e-akpakpan commented 3 years ago

@f-odhiambo, @mrjones-plip -- Need to troubleshoot the issue this morning with invalid Username and password for the OpenHIM Console at cop.app.medicmobile.org:9000. @f-odhiambo, you have done this install before so it would be great if you work with @mrjones-plip to get this done. Here are steps to follow:

mrjones-plip commented 3 years ago

@nsikak-e-akpakpan - I did not change the password and I'm unable to log into the server using the password you provided the other day - I have never tried logging before now, so no need to clear my browser cache. When I try now, I get this error:

An error occurred while attempting to connect to the OpenHIM Core on https://cop.app.medicmobile.org:8080.

The service may not be running or may not be accessible from you current location.

Additionally if Core is using a self-signed certificate, you may first need to instruct your browser to accept it. You can do so by accessing the following link.

Please contact your system administrator if the error persists or if the service was not accessible using the above link.

nsikak-e-akpakpan commented 3 years ago

@nsikak-e-akpakpan - I did not change the password and I'm unable to log into the server using the password you provided the other day - I have never tried logging before now, so no need to clear my browser cache. When I try now, I get this error:

An error occurred while attempting to connect to the OpenHIM Core on https://cop.app.medicmobile.org:8080. The service may not be running or may not be accessible from you current location. Additionally if Core is using a self-signed certificate, you may first need to instruct your browser to accept it. You can do so by accessing the following link. Please contact your system administrator if the error persists or if the service was not accessible using the above link.

I figured, Asked just in case. Please, work with @f-odhiambo to troubleshoot and if needed rerun the installation. Thanks

mrjones-plip commented 3 years ago

Looking at the output of docker ps|egrep 'PORT|8080' it looks OK:

CONTAINER ID   IMAGE                                         COMMAND                  CREATED        STATUS                                  PORTS                                                                                        NAMES
57a99090f255   hapiproject/hapi:v5.2.1                       "catalina.sh run"        38 hours ago   Up 38 hours                             0.0.0.0:3447->8080/tcp                                                                       hapi-fhir
eb10570322e1   jembi/openhim-core:5                          "docker-entrypoint.s…"   38 hours ago   Up 38 hours                             0.0.0.0:5000-5001->5000-5001/tcp, 0.0.0.0:5050-5052->5050-5052/tcp, 0.0.0.0:8080->8080/tcp   openhim-core

This is the same output as we saw yesterday - @nsikak-e-akpakpan can you confirm you were able to log in then?

mrjones-plip commented 3 years ago

for this step:

I think we should be sure we to check logs. I'm seeing interesting output from docker logs openhim-core that might be helpful, but I'm not sure:

2021-03-23T11:46:34.037Z [worker1] error: An httpServer clientError occured: Error: read ECONNRESET
2021-03-23T11:46:34.040Z [worker1] error: undefined
2021-03-23T11:46:34.126Z [worker1] error: An httpServer clientError occured: Error: Parse Error: Invalid method encountered
2021-03-23T11:46:38.227Z [worker1] info: No basic auth details supplied, trying next auth mechanism if any...
2021-03-23T11:46:38.228Z [worker1] info: Could NOT authenticate via TLS: undefined, trying next auth mechanism if any...
2021-03-23T11:46:38.268Z [worker1] info: No channel matched the request /
2021-03-23T11:46:38.269Z [worker1] info: The request, '/', is not authorised to access any channels.
2021-03-23T11:46:38.363Z [worker1] error: undefined
2021-03-23T11:46:38.373Z [worker1] error: An httpServer clientError occured: Error: read ECONNRESET
2021-03-23T11:46:38.373Z [worker1] error: undefined
2021-03-23T11:46:38.460Z [worker1] info: No basic auth details supplied, trying next auth mechanism if any...
2021-03-23T11:46:38.461Z [worker1] info: Could NOT authenticate via TLS: undefined, trying next auth mechanism if any...
2021-03-23T11:46:38.461Z [worker1] info: No channel matched the request /
2021-03-23T11:46:38.462Z [worker1] info: The request, '/', is not authorised to access any channels.
2021-03-23T12:23:50.440Z [worker1] info: API request made by undefined from cop.app.medicmobile.org:8080 is missing required API authentication headers, denying access
2021-03-23T12:27:12.646Z [worker1] info: test not found, trying next auth mechanism if any...
2021-03-23T12:27:12.647Z [worker1] info: Could NOT authenticate via TLS: undefined, trying next auth mechanism if any...
2021-03-23T12:27:12.649Z [worker1] info: No channel matched the request /fhir/Patient
2021-03-23T12:27:12.651Z [worker1] info: The request, '/fhir/Patient', is not authorised to access any channels.
2021-03-23T12:53:39.606Z [worker1] info: API request made by undefined from cop.app.medicmobile.org:8080 is missing required API authentication headers, denying access
mrjones-plip commented 3 years ago

After seeing that it's trying to GET in the dev console to https://cop.app.medicmobile.org:8080/authenticate/root@openhim.org, I saw that indeed you need to go to that URL and accept the invalid cert first.

After doing that, I get a new error "could not find user". You can see this with the curl test call:

➜  ~ curl -k https://cop.app.medicmobile.org:8080/authenticate/root@openhim.org
Could not find user by email root@openhim.org%
nsikak-e-akpakpan commented 3 years ago

After seeing that it's trying to GET in the dev console to https://cop.app.medicmobile.org:8080/authenticate/root@openhim.org, I saw that indeed you need to go to that URL and accept the invalid cert first.

After doing that, I get a new error "could not find user". You can see this with the curl test call:

➜  ~ curl -k https://cop.app.medicmobile.org:8080/authenticate/root@openhim.org
Could not find user by email root@openhim.org%

@mrjones-plip, @f-odhiambo - I have a few minutes to work on the server, if this ticket has not been resolved. Please respond

nsikak-e-akpakpan commented 3 years ago

Addressed issue with OpenHIM Console login as follows:

Issue resolved -- probably resource issue from the number of containers, now using a reduced number of containers. will monitor to see whether the issue reoccurs.

mrjones-plip commented 3 years ago

This issue seems to have returned so Iv'e reopened it:

CC @michaelkohn and @nsikak-e-akpakpan

mrjones-plip commented 3 years ago

I don't see any reboots of the containers since it was fixed 6 days ago - all containers have a STATUS of Up 6 days:

root@ip-10-35-4-237:~# docker ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED      STATUS      PORTS                                                                                        NAMES
80017db02f9b   hapiproject/hapi:v5.2.1      "catalina.sh run"        6 days ago   Up 6 days   0.0.0.0:3447->8080/tcp                                                                       hapi-fhir
363ee937143b   mongo:4.2                    "docker-entrypoint.s…"   6 days ago   Up 6 days   0.0.0.0:27017->27017/tcp                                                                     mongo-1
cbd0c5b16d17   mysql:5.7                    "docker-entrypoint.s…"   6 days ago   Up 6 days   0.0.0.0:3306->3306/tcp, 33060/tcp                                                            hapi-mysql
2c6b1adea6d9   jembi/openhim-console:1.14   "/docker-entrypoint.…"   6 days ago   Up 6 days   0.0.0.0:9000->80/tcp                                                                         openhim-console
f0dc59480dce   jembi/openhim-core:5         "docker-entrypoint.s…"   6 days ago   Up 6 days   0.0.0.0:5000-5001->5000-5001/tcp, 0.0.0.0:5050-5052->5050-5052/tcp, 0.0.0.0:8080->8080/tcp   openhim-core
ba6673971043   mongo:4.2                    "docker-entrypoint.s…"   6 days ago   Up 6 days   27017/tcp                                                                                    mongo-2
5759deba3c99   mongo:4.2                    "docker-entrypoint.s…"   6 days ago   Up 6 days   27017/tcp                                                                                    mongo-3
nsikak-e-akpakpan commented 3 years ago

@mrjones-plip, @f-odhiambo - Suggest restarting the containers and check if that fixes the problem. Try the following:

  1. Run -- sudo yarn docker:instant down -t docker core
  2. Verify docker containers are not running
  3. Run -- sudo yarn docker:instant up -t docker core
  4. Verify containers are running again
  5. Try login to console to see whether that fixes the issue
  6. Even if fixed, log issue with Jembi or OpenHIE to confirm whether the recommendation of scheduled shutdown and restart of the containers, per this closed issue on the topic
mrjones-plip commented 3 years ago

@nsikak-e-akpakpan - thanks so much for the tip! I ran this:

    cd /srv/chis/instant/                            # where the files are
    sudo yarn docker:instant down -t docker core     # to shutdown all containers
    docker ps                                        # ensure all containers are stopped
    sudo yarn docker:instant up -t docker core       # to start all  containers
    docker ps                                        # ensure all containers are started again
    docker logs openhim-core --follow

No luck, all problems still persist :(

nsikak-e-akpakpan commented 3 years ago

@mrjones-plip Thanks. I will take a look at the log later. Please do add the issue in Jembi and Instant-OpenHIE issue log for some guidance. This shows the OpenHIM Console heartbeat is up image

@nomulex, @f-odhiambo -- I suspect the issue has to do with the certificate. Is this something you can look into? I recall you mentioned this in a post before. I am strapped for the time needed to follow this trail to a solution. Here are some links that might help:

Thanks.

nomulex commented 3 years ago

@nomulex, @f-odhiambo -- I suspect the issue has to do with the certificate. Is this something you can look into? I recall you mentioned this in a post before. I am strapped for the time needed to follow this trail to a solution. Here are some links that might help:

@nsikak-e-akpakpan , I have pulled a letsencrypt SSL certificate on to the server. You can find it at

/etc/letsencrypt/live/cop.app.medicmobile.org/fullchain.pem

nsikak-e-akpakpan commented 3 years ago

@nomulex, @f-odhiambo -- Did any of you apply for the certificate? This issue has re-occurred. Can we please log this issue with Jembi and possibly find, someone, to let us know how to resolve this if we have not already done so? It seems, we may have to bypass OpenHIM for now and insert records directly into the FHIR server to move forward with the PoC. Thanks

cc: @michaelkohn -- we need help getting to hold of a contact an Jembi to work with us on this issue.

nomulex commented 3 years ago

Okay, If you gave me a heads up on where you need the certificate, I can install it. I have no clue of what and where you guys have installed on the box. ;) but I am happy to help you get unblocked. I need a little guidance though.

nsikak-e-akpakpan commented 3 years ago

@nomulex - Our issues are with the openHIM components (core and console). The core stores its data in MongoDB set (3 in the set). If you run: docker logs openhim-core you will see where the server is rejecting connections from the workers (processes). I also see whether it is having issues connecting with MongoDB. The only reference is the OpenHIM documentation here, which is not comprehensive. I think you can help with reaching out to Jembi Systems to find out what they have to help us resolve this issue. Thanks.

mrjones-plip commented 3 years ago

@nomulex - thanks very much for the Let's Encrypt work! I see the valid cert in /etc/letsencrypt per your comment above. As well, I see certbot is all set up to renew the certificates going forward.

@nsikak-e-akpakpan - Can you give the specific errors you're seeing that make you think this is a certificate issue? We've never used https externally to access the server, so this doesn't seem like the issue to me. The OpenHIM docs you cited show that we're using a non-existent certificate which hasn't been there all along, which to me further suggests this is not a certificate issue:

root@ip-10-35-4-237:~# docker exec -it openhim-core egrep -i 'keypath|cert' /usr/src/app/config/default.json
  "certificateManagement": {
    "watchFSForCert": false,
    "certPath": "/etc/letsencrypt/live/openhim.jembi.org/cert.pem",
    "keyPath": "/etc/letsencrypt/live/openhim.jembi.org/privkey.pem"

I do see some TLS errors (eg Could NOT authenticate via TLS: undefined, trying next auth mechanism if any...) , but those were there before and the fix was not to add a certificate that I know of.

However, since you set the server up, maybe you manually uploaded the certificate via the admin interface?

Per your mention of Mongo errors - can you add those here too?

Once we have all the info hopefully we can either debug ourselves or contact Jembi/Instant-OpenHIE as needed.

nsikak-e-akpakpan commented 3 years ago

@mrjones-plip I suggest we do both - contact Jembi while we continue debugging. I mentioned the certificate issue, because of the TLS errors and the server refusing the connection. You can test out by resetting the containers to get access to the console and adding the certificate Alex created. Meant to do that over the weekend, but could not get around to it. Nsikak

mrjones-plip commented 3 years ago

@nsikak-e-akpakpan - Sounds good! I tried to get to it this week but ran out of time. I'll review the docs over at openhie's instant repo and then open an issue with them early next week.

Stay tuned!

mrjones-plip commented 3 years ago

Hopefully this has been solved! Fix was to redo the deployment using docker-compse and to redo some of the initialization scripts originally in Instant OpenHIE. All changes are now in this repo.