Docker Networking: implement proxy container and define docker networks

br2490 commented 6 years ago

I would like to see some development around this. It deprecates the need for editing your "etc/hosts" or adding "extra_hosts" - please see latest bcol migration compose for working example - granted this was done without some documentation so it may need a fewwwwwww tweaks. (https://github.com/BarnardArchives/barnard_isle_docker)

I'll also leave the result of this example migration on/up for the day at http://belle.benjaminrosner.com (it's just a VM on my workstation doing this). There are still a few things to note as issue - like some proxying of services up to the user layer (looking at you djatoka) -

I'm wondering if in the next stages we could create a small NGINX proxy to sit in front of the entire stack - and on the only docker publicly-accessible network - so that all comms go through NGINX. It could be secure with certs. I wonder if that also means there is additional opportunity to "load balance" with containers (i.e., roundrobin the heck of out multiple instances of the same container in nginx). So many possibilities I LOVE IT!

DEVEL:

[x] delineation between an internal and external network, where internal is only accessible to the stack itself, and external is for wide access. this is
[x] add a new service. Maybe called proxy for testing and name TBD later.
- [x] use image NGINX:LATEST
- [x] generate generic service-level definition of all stack components as upstreams (LB)
- [x] proxy routes for services required for external visitors
[x] On image isle-apache:
- [x] create generic isle and isle-ssl confs.

TEST:

[x] certs
[ ] cert bot
[ ] snakeoil to secure intra-stack comms too? meh.
[ ] security

A test instance is running at https://belle.benjaminrosner.com nb: fgs still will not load, so the stack is incomplete in the truest sense, but the site works, fedora, solr, etc.

g7morris commented 6 years ago

@br2490 This is something I've been thinking about for awhile now re NGINX proxy. I'm wondering how much could be gained by using something like this https://traefik.io/ (nginx proxy designed for these type of setups) Do you know anything about it? I'm curious what benefits it could have for SSL, Djatoka etc. and helping to better route things in general.

McFateM commented 6 years ago

My $0.02…. Traefik and Nginx are bedrock elements of the docker4drupal stuff that I’ve been playing with lately. I’m sure I don’t understand it fully, and I have not tried applying either to ISLE, nor have I done any SSL work in Nginx yet, but I really like what I’ve seen in docker4drupal thus far. I have successfully applied d4d to building a Dockerized version of two Drupal 8 sites that I develop and maintain. Can’t say it’s better than Apache and vhosts, but it’s certainly not worse. In my own experience, Apache + Vhosts = potential for headaches and TROUBLE. And SSL just exacerbates that. Traefix and Nginx have been virtually trouble-free for me thus far.

-Mark M.

From: Gavin Morris notifications@github.com Reply-To: Islandora-Collaboration-Group/ISLE reply@reply.github.com Date: Friday, February 16, 2018 at 2:27 PM To: Islandora-Collaboration-Group/ISLE ISLE@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [Islandora-Collaboration-Group/ISLE] Docker networking (#103)

@br2490https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_br2490&d=DwMFaQ&c=HUrdOLg_tCr0UMeDjWLBOM9lLDRpsndbROGxEKQRFzk&r=PQglHQe-EzyZqJOuOVcmU0OZ6bg-89msSPuqyNlQr28&m=_ZaTiIlxD9m-pMw6EKIZpQtrsz-udKtZrHpfXdIXiEc&s=gDLUSqEL3h_5KShofVM9s4ceCwMUu-TbXSakZHNjsKs&e= This is something I've been thinking about for awhile now re NGINX proxy. I'm wondering how much could be gained by using something like this https://traefik.io/https://urldefense.proofpoint.com/v2/url?u=https-3A__traefik.io_&d=DwMFaQ&c=HUrdOLg_tCr0UMeDjWLBOM9lLDRpsndbROGxEKQRFzk&r=PQglHQe-EzyZqJOuOVcmU0OZ6bg-89msSPuqyNlQr28&m=_ZaTiIlxD9m-pMw6EKIZpQtrsz-udKtZrHpfXdIXiEc&s=jVNWQi8s9l2ZTqwYq7VwN49lcaA00eYnVL7e1HqlPgI&e= (nginx proxy designed for these type of setups) Do you know anything about it? I'm curious what benefits it could have for SSL, Djatoka etc. and helping to better route things in general.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Islandora-2DCollaboration-2DGroup_ISLE_issues_103-23issuecomment-2D366349123&d=DwMFaQ&c=HUrdOLg_tCr0UMeDjWLBOM9lLDRpsndbROGxEKQRFzk&r=PQglHQe-EzyZqJOuOVcmU0OZ6bg-89msSPuqyNlQr28&m=_ZaTiIlxD9m-pMw6EKIZpQtrsz-udKtZrHpfXdIXiEc&s=I6COOv5z_vbVB4b4LQ7pG5_6dUMuUmGogO1YYPC3Egk&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AIFIwRZiwQj66vxyc8HKJEJSYyO-2D9gKsks5tVeStgaJpZM4SIRRg&d=DwMFaQ&c=HUrdOLg_tCr0UMeDjWLBOM9lLDRpsndbROGxEKQRFzk&r=PQglHQe-EzyZqJOuOVcmU0OZ6bg-89msSPuqyNlQr28&m=_ZaTiIlxD9m-pMw6EKIZpQtrsz-udKtZrHpfXdIXiEc&s=L6vfIrhFCDmpKO75ByTj79gr7ej4Vp-71G_3BIAAnAo&e=.

br2490 commented 6 years ago

Status: in development need early review @McFateM @g7morris if https://belle.benjaminrosner.com/ && http://belle.benjaminrosner.com/ work as expected externally

Development thoughts

[x] delineation between an internal and external network, where internal is only accessible to the stack itself, and external is for wide access. this is
[x] add a new service. Maybe called proxy for testing and name TBD later.
- [x] use image NGINX:LATEST
- [x] generate generic service-level definition of all stack components as upstreams (LB)
- [x] proxy routes for services required for external visitors
[x] On image isle-apache:
- [x] create generic isle and isle-ssl confs.
[x] @g7morris could you note additional areas that should be coming through this proxy that are currently proxied via something apache in the stack? I have a definition for SOLR but that's actually just an internal-to-the-stack service, right?

hitlist

[x] add stack network, add all services.
[x] public ingress network created and proxy added to it
[x] ports 80, 443 exposed; opened on router.
[x] SSL certs (oh goodness)
[x] generalized configs for apache; nginx config to proxy upstreams for apache, fedora, solr (unused).
[x] TEST RUNNING

The development proxy container is running - 17 FEB 2018 noonish.

br2490 commented 6 years ago

https://www.ssllabs.com/ssltest/analyze.html?d=belle.benjaminrosner.com for default ciphers? @hamhpc @g7morris

g7morris commented 6 years ago

@br2490 Wow that's a lot of progress. Nice!

Update: Ignore the first question, I went too quickly and didn't read the post above that clearly indicated https for the homepage was available. Apologies for wasting time.

Q: Any chance you could put the setup in a private repo for review? (Would be greatly appreciated)

A: Typically for the proxies, really only djatoka needs to be proxied but I think I also include some Fedora information to be safe:

Note 1: Using the http / 8080 as an example below. substitute https:// and 8443

Note 2: I think the previous use of fedora as the domain name below may be incorrect and might want to be avoided. Endusers may want to use https://fedora.domainname.extension instead.

Note 3: Note 2 then begs the question is the use of solr for search on one environment will it work in a multi-network setup. Clearly trying to avoid one solr being used for other domains. I think this is avoided due to clearly separate networks but perhaps some diagramming out for networks could help?

I'm VERY curious now how you used Traefik to do all this. I'm starting to think it's inclusion might be necessary in multi-domain / multi-environment setups aka running production, stage and dev with respective domains on one ISLE Host Server.

Example of Vhosts below proxied for adore-djatoka:

Fedora vhost ProxyPass /adore-djatoka http://fedora:8080/adore-djatoka ProxyPassReverse /adore-djatoka http://fedora:8080/adore-djatoka

br2490 commented 6 years ago

@g7morris see our repo for working example.

I updated the readme but it is still out of date -- it doesn't mention the proxy, certificates, etc. However the docker-compose is rather simple for the proxy.

Now would be the time to get letsencrypt bot going (#44)

Also circling back to Diego and caching responses from loc.gov (or sorry, whatever the correct address is for validation) should be implemented here. network alias the proxy to the appropriate URL, create a proxypass to the real site, set a cache of X duration, rinse, repeat. Or just serve them statically (i'm not a lawyer)

@g7morris we should absolute chat about how this works, and all.

g7morris commented 6 years ago

@br2490 I think this ticket can be closed no? Aren't we doing all this now?

Islandora-Collaboration-Group / ISLE

Docker Networking: implement proxy container and define docker networks #103