freedomofpress / securethenews

An automated scanner and web dashboard for tracking TLS deployment across news organizations
https://securethe.news
GNU Affero General Public License v3.0
102 stars 25 forks source link

Allow local spin-up of securethenews with prod #58

Closed msheiny closed 7 years ago

msheiny commented 7 years ago

Allow for tracking of example playbook locally in the STN repo and for fast-tracking stand-up of an environment that mimics prod.

Need to work with @conorsch to get kvm working and need to get a guinea pig to test against OSX

conorsch commented 7 years ago

Oh snap, this looks good! Will spin a libvirt VM locally, one moment...

conorsch commented 7 years ago

Failed to install packages:

Installing setuptools, pip, wheel...done.
. ./venv/bin/activate; \
pip install -r ../../django_stack/requirements.txt 
Collecting ansible==2.2.0.0 (from -r ../../django_stack/requirements.txt (line 1))
Collecting cffi==1.8.3 (from -r ../../django_stack/requirements.txt (line 2))
  Using cached cffi-1.8.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting cryptography==1.5.2 (from -r ../../django_stack/requirements.txt (line 3))
Collecting enum34==1.1.6 (from -r ../../django_stack/requirements.txt (line 4))
  Using cached enum34-1.1.6-py2-none-any.whl
Collecting idna==2.1 (from -r ../../django_stack/requirements.txt (line 5))
  Using cached idna-2.1-py2.py3-none-any.whl
Collecting ipaddress==1.0.17 (from -r ../../django_stack/requirements.txt (line 6))
  Using cached ipaddress-1.0.17-py2-none-any.whl
Collecting Jinja2==2.8 (from -r ../../django_stack/requirements.txt (line 7))
  Using cached Jinja2-2.8-py2.py3-none-any.whl
Collecting MarkupSafe==0.23 (from -r ../../django_stack/requirements.txt (line 8))
Collecting molecule==1.12.6 (from -r ../../django_stack/requirements.txt (line 9))
  Using cached molecule-1.12.6-py2-none-any.whl
Collecting paramiko==2.0.2 (from -r ../../django_stack/requirements.txt (line 10))
  Using cached paramiko-2.0.2-py2.py3-none-any.whl
Collecting pkg-resources==0.0.0 (from -r ../../django_stack/requirements.txt (line 11))
  Could not find a version that satisfies the requirement pkg-resources==0.0.0 (from -r ../../django_stack/requirements.txt (line 11)) (from versions: )
No matching distribution found for pkg-resources==0.0.0 (from -r ../../django_stack/requirements.txt (line 11))
Makefile:10: recipe for target 'pip' failed

What's up with the pkg-resources==0.0.0 line in requirements? Also, isn't the Makefile assuming that you've cloned down django_stack adjacent to the STN project dir? Wouldn't it be better to clone the django-stack role as part of the makefile setup?

msheiny commented 7 years ago

What's up with the pkg-resources==0.0.0 line in requirements?

Oh shit - i noticed that too, just yanked it out of the requirements.txt file, need to push that up to master

Wouldn't it be better to clone the django-stack role as part of the makefile setup?

So clone it under a sub-directory? I was thinking next I want to circumvent the git pull-down tasks and sling the local repo directory to the guest VM. Didn't want to also copy all django_stack code... I guess it's not a huge deal though.... hmmmm

conorsch commented 7 years ago

So clone it under a sub-directory? I was thinking next I want to circumvent the git pull-down tasks and sling the local repo directory to the guest VM. Didn't want to also copy all django_stack code

We already need to clone done all the dependent roles, e.g. postgres and nodejs, so django-stack is just a needle in the haystack, really.

msheiny commented 7 years ago

We already need to clone done all the dependent roles, e.g. postgres and nodejs, so django-stack is just a needle in the haystack, really.

Think 54b9a35 resolves your concerns @conorsch - how were you able to test this the other day if you are off the virtualbox wagon?

msheiny commented 7 years ago

@conorsch - can you review? @garrettr - i'd like to make this platform agnostic but i no longer run virtualbox under my grsec kernel so i have to run it against libvirt + kvm. We can probably tweak the makefile a bit to detect platform and run molcule against a specific provider. If you want to mess with that, kewl... if not, i'll just hang tight for @conorsch to confirm he can also get this running under kvm for now.

conorsch commented 7 years ago

Whoops, missed this. Testing now!

conorsch commented 7 years ago

Going to keep hacking away at this one, will take some time to build/destroyrebuild. Off the cuff I'm a bit concerned about the two separate environments, viz. vagrant vs. molecule, but we can always rip out the raw Vagrant config later on. We also have two Makefiles now, as of these changes.

Will be testing with libvirt and report back, so we have a sane env for testing changes locally.

msheiny commented 7 years ago

I'm a bit concerned about the two separate environments, viz. vagrant vs. molecule, but we can always rip out the raw Vagrant config later on. We also have two Makefiles now, as of these changes.

Right totally. I also share that concern. I'm hoping we can purge the other folder and merge these changes into the top level Makefile. Only issue is that our layout isn't really cross-compatible yet so if I destroy that it would hamper @garrettr 's current testing workflow. I think the best compromise would be to also define virtualbox as a provider under the molecule config and then have the Makefile be smart enough to filter to the right provider.

conorsch commented 7 years ago

LGTM, @msheiny! I had to add an exception for the self-signed cert for browsing over HTTPS, but that's to be expected. I'm happy with this going in as-is, we'll tweak with future updates as necessary.