Migrates SSH Observatory datastore from MongoDB to Postgres

[claudijd]

This is a work in progress to put together a postgres backend replacement for mongodb. It's far from done, but a PR is the best place to share with others and get their feedback.

[claudijd]

With the latest push, this should start passing again. I will also be doing some bench sanity checks to make sure it still works end to end before landing this change.

[claudijd]

Ok, so finally able to task scans properly, so we're getting there, next up, troubleshooting the minor changes we made but forgot to adjust the interfaces for.

[claudijd]

Instructions for alpha testers...

Requirements 1.) install 'jq' (OPTIONAL) 2.) python requests (assuming you use python client)

Stand Up Infrastructure 1.) clone this repo 2.) checkout this branch 3.) docker-compose up --build

Run a scan from an example client 4a.) python examples/client.py <= if you don't have jq installed 4b.) python examples/client.py | jq <= if you have jq installed

Provide feedback 5.) let me know whether it worked or not and if no why (also include any other feedback you feel is relevant to improving this)

[pwnbus]

Seems to work just fine on my end! Nice!

[claudijd]

@pwnbus proposed inheriting to save build time on rubygems, the latest commit speeds things up substantually for build time. went from 1min 30sec builds to < 10sec builds (wow!) after the cache is built once.

[caggle]

Instructions worked great for me, no issues with or without jq.

I did notice though, when you scan a system with 22/tcp is closed, currently the scan goes on forever? And worker instance borks some time when this happens.

[SCMGuruLLC]

I'm running Ubuntu Server 16.04.3 LTS 32-bit.

I ran: 3.) docker-compose up --build

And got: ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running? If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

I ran: sudo docker-compose up --build

And got: Creating network "sshscanapi_default" with the default driver Building base Step 1/8 : FROM ruby:latest AS base ERROR: Service 'base' failed to build: Error parsing reference: "ruby:latest AS base" is not a valid repository/tag: invalid reference format

[claudijd]

@cag I believe ssh_scan has logic to handle the socket timeout and will drop the scan result into an error state. Unfortunately, the clients are very unsophisticated, mainly because they are for demo purposes and don't have logic to capture that error condition. That said, you should be able to verify this by accessing this route after attempting the scan...

https://sshscan.rubidus.com/api/v1/scan/results?uuid=db49ba74-8482-4c30-9cbe-66f4ae62683b

You just need to replace the UUID in the url to the UUID the client says it got for that scan. We will then return everything we have for that scan, but if the socket timed out, it will be very little, but it should be there. In other words, I believe this is working as expected, please let me know if it doesn't behave this way for you.

[claudijd]

@scmaccal the sudo/non-sudo behavior is simply a function of your current user not having docker privs. If you add that user to the docker group, I think that will solve the problem.

As for the second issue, I'm wondering if it's a docker version issue, would you mind sharing the output of the following:

docker-compose version docker version

I'm assuming your version of docker doesn't support the AS capability, but I'd like to compare to be sure.

[SCMGuruLLC]

sudo docker-compose version docker-compose version 1.8.0, build unknown docker-py version: 1.9.0 CPython version: 2.7.12 OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016

sudo docker version Client: Version: 1.13.1 API version: 1.26 Go version: go1.6.2 Git commit: 092cba3 Built: Thu Nov 2 20:40:26 2017 OS/Arch: linux/386

Server: Version: 1.13.1 API version: 1.26 (minimum version 1.12) Go version: go1.6.2 Git commit: 092cba3 Built: Thu Nov 2 20:40:26 2017 OS/Arch: linux/386 Experimental: false

On Wed, Jan 17, 2018 at 2:06 PM, Jonathan Claudius <notifications@github.com

wrote:

@scmaccal https://github.com/scmaccal the sudo/non-sudo behavior is simply a function of your current user not having docker privs. If you add this person to the docker group, I think that will solve the problem.

As for the second issue, I'm wondering if it's a docker version issue, would you mind sharing the output of the following:

docker-compose version docker version

I'm assuming your version of docker doesn't support the AS capability, but I'd like to compare to be sure.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mozilla/ssh_scan_api/pull/121#issuecomment-358408913, or mute the thread https://github.com/notifications/unsubscribe-auth/ABYY7iQIHzGII5_Fl2-sqd0uE6vQiUA4ks5tLkS4gaJpZM4RYU4w .

-- Scott

[claudijd]

@scmaccal I'm running docker-compose 1.18.0 and docker 17.12.0-ce. I'm wondering if upgrading both of those on your system will fulfill the FROM/AS behavior and start working.

[SCMGuruLLC]

I did some research and in order for me to install the latest version of Docker, I need to be running a 64-bit OS. I'll try this next. I suspect the problem will go away. We'll see.

On Wed, Jan 17, 2018 at 2:35 PM, Jonathan Claudius <notifications@github.com

wrote:

@scmaccal https://github.com/scmaccal I'm running docker-compose 1.18.0 and docker 17.12.0-ce. I'm wondering if upgrading both of those on your system will fulfill the FROM/AS behavior and start working.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mozilla/ssh_scan_api/pull/121#issuecomment-358417540, or mute the thread https://github.com/notifications/unsubscribe-auth/ABYY7vQGJn4fedorddSqAFPYCAnewGNBks5tLktHgaJpZM4RYU4w .

-- Scott

[SCMGuruLLC]

I added the official repositories from Docker and installed the latest version of the docker and docker-compose packages. Everything works fine now on Ubuntu Server 16.04.3 LTS 64-bit.

I recommend adding to the build requirements for Docker that only a 64-bit OS and the latest version of Docker will work. I think language about how the official repositories provided by Docker are best would be helpful too.

Given that by default the Docker packages don't add a user to an elevated user group on Ubuntu and probably other Unix-like distributions, I recommend language about running sudo before the docker-compose command.

Under Stand Up Infrastructure for number 1, I recommend adding: git clone https://github.com/mozilla/ssh_scan_api.git For number 2, add: git fetch git checkout postgres_sql_prototype

I'm willing to maintain any building or other documentation for this or stable project. Just point me in the right direction. :)

On Jan 17, 2018 3:43 PM, "Scott C. MacCallum" maccallum.c.scott@gmail.com wrote:

I did some research and in order for me to install the latest version of Docker, I need to be running a 64-bit OS. I'll try this next. I suspect the problem will go away. We'll see.

On Wed, Jan 17, 2018 at 2:35 PM, Jonathan Claudius < notifications@github.com> wrote:

@scmaccal https://github.com/scmaccal I'm running docker-compose 1.18.0 and docker 17.12.0-ce. I'm wondering if upgrading both of those on your system will fulfill the FROM/AS behavior and start working.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mozilla/ssh_scan_api/pull/121#issuecomment-358417540, or mute the thread https://github.com/notifications/unsubscribe-auth/ABYY7vQGJn4fedorddSqAFPYCAnewGNBks5tLktHgaJpZM4RYU4w .

-- Scott

[claudijd]

@scmaccal I have a doc for the typical docker infra setup, which is maintained on the GitHub wiki here (https://github.com/mozilla/ssh_scan_api/wiki/Deploying-ssh_scan_api-using-docker-compose). I'll probably add these additions there and then just ask people to follow that guide (since this PR will likely be deployed by EOW). Thanks for the feedback, that's really helpful to avoid future issues for people trying it out.

[claudijd]

@scmaccal the wiki instructions have been updated. For anyone wishing to test out this branch, they simply need to follow the instructions here and replace the git checkout master with git checkout postgres_sql_prototype and to let me know if they have any problems.

[claudijd]

@scmaccal you may have gotten a collaboration request on this repo. It was my attempt to list you as a named reviewer on this PR, but I guess that's not possible without giving you write access. I ended up canceling it because it didn't give me the desirer effect. I think your comments and help here have been excellent, thanks again for the help with this!

[SCMGuruLLC]

You're welcome. Happy to help.

Scott

On Jan 17, 2018 8:53 PM, "Jonathan Claudius" notifications@github.com wrote:

@scmaccal https://github.com/scmaccal you may have gotten a collaboration request on this repo. It was my attempt to list you as a named reviewer on this PR, but I guess that's not possible without giving you write access. I ended up canceling it because it didn't give me the desire effect. I think your comments and help here have been excellent, thanks again for the help with this!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mozilla/ssh_scan_api/pull/121#issuecomment-358511699, or mute the thread https://github.com/notifications/unsubscribe-auth/ABYY7rgPKS1CucFgmXmtF__Yyccd8OmOks5tLqQWgaJpZM4RYU4w .

[caggle]

@claudijd Thanks. Yes I didn't realise the logic to handle socket timeout was actually in the proper client (ssh_scan).

Viewing the scan result for a scan attempted on a closed port (e.g. http://127.0.0.1:8000/api/v1/scan/results?uuid=0c435d0d-ddae-494d-a01d-b8c78bb5a01d) returns "status": "ERRORED", which is probably the expected result.

[claudijd]

Just an FYI: although this PR worked on the bench, in practice under a multi-threaded load, this PG port doesn't operate very well and is entirely too unstable to run in any predictable way. That said, I'm probably going to be rolling back these changes on the production server today and I'll be changing my strategy to leverage a better supported SQL abstraction, such as ActiveRecord. I've got the first commit of that effort started in https://github.com/mozilla/ssh_scan_api/pull/137 and I'll be shooting for a rough target of EOW to land the ActiveRecord port.