varnish pod crashloopback

bernardmaltais commented 4 years ago

Trying to make use of the recently updated varnish template result in a pod crashloopback with the following error:

Error:
--
Sun, Nov 22 2020 8:38:21 am | Message from VCC-compiler:
Sun, Nov 22 2020 8:38:21 am | Backend host '"drupal-wxt-varnish-drupal"' could not be resolved to an IP address:
Sun, Nov 22 2020 8:38:21 am | Name or service not known
Sun, Nov 22 2020 8:38:21 am | (Sorry if that error message is gibberish.)
Sun, Nov 22 2020 8:38:21 am | ('/etc/varnish/default.vcl' Line 4 Pos 11)
Sun, Nov 22 2020 8:38:21 am | .host = "drupal-wxt-varnish-drupal";
Sun, Nov 22 2020 8:38:21 am | ----------###########################-
Sun, Nov 22 2020 8:38:21 am |  
Sun, Nov 22 2020 8:38:21 am |  
Sun, Nov 22 2020 8:38:21 am | In backend specification starting at:
Sun, Nov 22 2020 8:38:21 am | ('/etc/varnish/default.vcl' Line 3 Pos 1)
Sun, Nov 22 2020 8:38:21 am | backend default {
Sun, Nov 22 2020 8:38:21 am | #######----------
Sun, Nov 22 2020 8:38:21 am |  
Sun, Nov 22 2020 8:38:21 am | Running VCC-compiler failed, exited with 2
Sun, Nov 22 2020 8:38:21 am | VCL compilation failed

bernardmaltais commented 4 years ago

I think it has to do with the host name it is trying to find. It is looking for drupal-wxt-varnish-drupal but that name does not exist. On the other hand drupal-wxt-varnish does exist.

I manually created a service with the name drupal-wxt-varnish-drupal and the pod came up clean. So I suspect an issue with the way the service name in the varnish helm chart is computed to the wrong value.

bernardmaltais commented 4 years ago

Also, looking at the template code I am not sure varnish would actually get used as I am not finding code that would reconfigure the ingress to use varnish instead of nginx. Is it possible this just deploy varnish without integrating it in the flow between the user and nginx? https://github.com/drupalwxt/helm-drupal/blob/4d82a1295ba62f980d9954d7dbc70a4a0127261c/drupal/templates/ing/drupal.yaml

sylus commented 4 years ago

Ah sorry this was pushed half done, and it was supposed to be updated last week from our working internal.

I have just synced it right now in this P.R. and waiting for @zachomedia to approve it :D

https://github.com/drupalwxt/helm-drupal/pull/40/files

bernardmaltais commented 4 years ago

Well well well... after digging a bit more in the varnish chart I figured there is a very important value that actually need to be passed within the varnish: section of the helm-drupal char values.yaml... the varnishConfigContent value. After setting it to the appropriate values things appear to deploy properly. The default ingress is still not pointing to varnish so no traffic actually goes through it but it does deploy properly.

## Configuration values for the Varnish dependency sub-chart
## ref: https://github.com/StatCan/charts/blob/master/stable/varnish/README.md
varnish:
  enabled: true
  varnishd:
    image: varnish
    tag: 6.4.0
    imagePullPolicy: IfNotPresent
  service:
    type: ClusterIP
    port: 80
  resources: {}
  #  requests:
  #    memory: "512Mi"
  #    cpu: "100m"
  #  limits:
  #    memory: "1Gi"
  #    cpu: "500m"
  nodeSelector: {}
  tolerations: []
  affinity: {}
  varnishConfigContent: |
      vcl 4.0;
      # Assumed 'drupal' host, this can be docker servicename
      backend default {
          .host = "drupal-wxt-nginx";
          .port = "80";
      }

      sub vcl_recv {
        # Do not cache these paths.
        if (req.url ~ "^/status\.php$" ||
            req.url ~ "^/cron\.php.*$" ||
            req.url ~ "^/update\.php$" ||
            req.url ~ "^/ooyala/ping$" ||
            req.url ~ "^/admin/build/features" ||
            req.url ~ "^/info/.*$" ||
            req.url ~ "^/flag/.*$" ||
            req.url ~ "^.*/ajax/.*$" ||
            req.url ~ "^.*/ahah/.*$") {
            return (pass);
        }

        if (req.http.Authorization || req.http.Authenticate) {
          return (pass);
        }
      }

      sub vcl_recv {
        # Strip anchors, server doesn't need it.
        if ( req.url ~ "\#" ) {
          set req.url = regsub(req.url, "\#.*$", "");
        }
      }

      sub vcl_recv {
        # Strip a trailing ? if it exists
        if ( req.url ~ "\?$" ) {
          set req.url = regsub(req.url, "\?$", "");
        }
      }

      sub vcl_backend_response {
        if (bereq.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf|ogg|ogm|opus|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") {
          unset beresp.http.set-cookie;
        }
        return (deliver);
      }

      sub vcl_backend_response {
        # Set 60min cache if unset for static files
        if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") {
          set beresp.ttl = 3600s; # Important, you shouldn't rely on this, SET YOUR HEADERS in the backend
          set beresp.uncacheable = true;
          return (deliver);
        }
        return (deliver);
      }

      sub vcl_backend_response {
        # Allow stale content, in case the backend goes down.
        # make Varnish keep all objects for 6 hours beyond their TTL
        set beresp.grace = 6h;
      }

I think this value should not have to be provided in the values file and a proper default for helm-drupal should be baked into the chart to make it work out of the box...

Varnish appear to be very much a work in progress?

sylus commented 4 years ago

Ah @bernardmaltais think you just posted after what I just posted :P

There is a P.R. with all this working just waiting on @zachomedia to approve it

https://github.com/drupalwxt/helm-drupal/pull/40/files

We do also bake all of this into the chart so shouldn't be anything you have to do :D

bernardmaltais commented 4 years ago

@sylus Nice. I was wondering about that. This gave me a chance to poke at the bear so to speak to make it growl ;-) I will look forward to the merge for the update. Many thanks!

sylus commented 4 years ago

Hey @bernardmaltais is now merged :D

https://github.com/drupalwxt/helm-drupal/pull/40

Let us know how it works for you, i'll spin it up in my kind cluster and test it as well sometime today.

sylus commented 4 years ago

1) kind create cluster --config kind.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  # add a mount from /path/to/my/files on the host to /files on the node
  extraMounts:
  - hostPath: /tmp/drupal/shared/drupal-public
    containerPath: /mnt/drupal/drupal-public
  - hostPath: /tmp/drupal/shared/drupal-private
    containerPath: /mnt/drupal/drupal-private

2) helm install drupal -f values-hostpath-kind.yaml --set varnish.enabled=true .

NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
default              drupal-7ff76d6f74-s6zn2                      1/1     Running   0          83s
default              drupal-mysql-54c79c5658-6lggk                1/1     Running   0          83s
default              drupal-nginx-787db5cf7c-2kw2h                1/1     Running   0          83s
default              drupal-site-install-6ddxp                    1/1     Running   0          82s
default              drupal-varnish-7b75c46d6c-4nq9h              1/1     Running   0          83s

3) Example of output

DB available
 Drupal version : 9.0.8                                 
 Site URI       : http://default                        
 DB driver      : mysql                                 
 DB hostname    : drupal-mysql                          
 DB port        : 3306                                  
 DB username    : wxt                                   
 DB name        : wxt                                   
 PHP binary     : /usr/local/bin/php                    
 PHP config     : /usr/local/etc/php/php.ini            
 PHP OS         : Linux                                 
 Drush script   : /var/www/vendor/drush/drush/drush     
 Drush version  : 10.3.6                                
 Drush temp     : /tmp                                  
 Drush configs  : /var/www/vendor/drush/drush/drush.yml 
 Drupal root    : /var/www/html                         
 Site path      : sites/default                         

 You are about to:
 * DROP all tables in your 'wxt' database.

 // Do you want to continue?: yes.                                              

 [notice] Starting Drupal installation. This takes a while.
 [notice] Performed install task: install_select_language
 [notice] Performed install task: install_select_profile
 [notice] Performed install task: install_load_profile
 [notice] Performed install task: install_verify_requirements
 [notice] Performed install task: install_verify_database_ready
 [notice] Performed install task: install_base_system
 [notice] Performed install task: install_bootstrap_full
 [notice] Performed install task: install_profile_modules
 [notice] Performed install task: install_profile_themes
 [notice] Performed install task: install_install_profile
 [notice] Translations imported: 9187 added, 0 updated, 0 removed.
 [notice] Performed install task: install_import_translations
 [notice] Performed install task: wxt_extension_configure_form
 [notice] Performed install task: install_configure_form
 [notice] metatag.metatag_defaults.global rewritten by wxt_ext_metatag
 [notice] metatag.metatag_defaults.node rewritten by wxt_ext_metatag
 [notice] metatag.metatag_defaults.taxonomy_term rewritten by wxt_ext_metatag
 [notice] metatag.metatag_defaults.user rewritten by wxt_ext_metatag
 [notice] core.entity_view_display.node.blog_post.default rewritten by wxt_ext_blog
 [notice] core.entity_form_display.node.blog_post.default rewritten by wxt_ext_blog
 [notice] views.view.blog rewritten by wxt_ext_blog
 [notice] node.type.blog_post rewritten by wxt_ext_blog
 [notice] field.field.node.blog_post.field_blog_tags rewritten by wxt_ext_blog
 [notice] field.storage.node.field_blog_tags rewritten by wxt_ext_blog
 [notice] field.field.node.blog_post.field_blog_comments rewritten by wxt_ext_blog
 [notice] menu_breadcrumb.settings rewritten by wxt_ext_breadcrumb
 [notice] image.style.linkit_result_thumbnail rewritten by wxt_ext_editor
 [notice] linkit.linkit_profile.default rewritten by wxt_ext_editor
 [notice] editor.editor.rich_text rewritten by wxt_ext_editor
 [notice] filter.format.rich_text rewritten by wxt_ext_editor
 [notice] fontawesome.settings rewritten by wxt_ext_fontawesome
 [notice] group.settings rewritten by wxt_ext_group
 [notice] pathauto.pattern.group_content rewritten by wxt_ext_group
 [notice] field.storage.node.layout_builder__layout rewritten by wxt_ext_layout
 [notice] layout_library.layout.page_two_column rewritten by wxt_ext_layout
 [notice] field.storage.node.layout_selection rewritten by wxt_ext_layout
 [notice] field.storage.node.layout_builder__translation rewritten by wxt_ext_layout
 [notice] core.entity_view_display.node.landing_page.full rewritten by wxt_ext_landing_page
 [notice] field.field.node.landing_page.layout_selection rewritten by wxt_ext_landing_page
 [notice] core.entity_form_display.node.landing_page.default rewritten by wxt_ext_landing_page
 [notice] field.field.node.landing_page.layout_builder__layout rewritten by wxt_ext_landing_page
 [notice] field.field.node.landing_page.layout_builder__translation rewritten by wxt_ext_landing_page
 [notice] field.field.node.page.layout_selection rewritten by wxt_ext_page
 [notice] core.entity_view_display.node.page.default rewritten by wxt_ext_page
 [notice] node.type.page rewritten by wxt_ext_page
 [notice] core.entity_form_display.node.page.default rewritten by wxt_ext_page
 [notice] core.entity_form_display.node.landing_page.default rewritten by wxt_ext_queue
 [notice] views.view.blocks rewritten by wxt_ext_queue
 [notice] system.theme rewritten by wxt_ext_theme
 [notice] node.settings rewritten by wxt_ext_theme
 [notice] user.settings rewritten by wxt_ext_user
 [notice] field.field.node.webform.body rewritten by wxt_ext_webform
 [notice] core.entity_view_display.node.webform.teaser rewritten by wxt_ext_webform
 [notice] core.entity_form_display.node.webform.default rewritten by wxt_ext_webform
 [notice] field.field.node.webform.webform rewritten by wxt_ext_webform
 [notice] field.storage.node.webform rewritten by wxt_ext_webform
 [notice] node.type.webform rewritten by wxt_ext_webform
 [notice] core.entity_view_display.node.webform.default rewritten by wxt_ext_webform
 [notice] Translation file not found: http://ftp.drupal.org/files/translations/9.x/drupal/drupal-9.0.8.fr.po.
 [notice] Translation file not found: http://ftp.drupal.org/files/translations/9.x/drupal/drupal-9.0.8.fr.po.
 [notice] Translation file not found: http://ftp.drupal.org/files/translations/9.x/drupal/drupal-9.0.8.fr.po.
 [notice] Translation file not found: http://ftp.drupal.org/files/translations/9.x/drupal/drupal-9.0.8.fr.po.
 [notice] Translation file not found: http://ftp.drupal.org/files/translations/9.x/drupal/drupal-9.0.8.fr.po.
 [notice] Performed install task: wxt_install_extensions
 [notice] Performed install task: install_finish_translations
 [notice] Cron run completed.
 [notice] Performed install task: install_finished
 [notice] Performed install task: wxt_import_language_config
 [success] Installation complete.
real    2m 39.86s
user    1m 37.26s
sys     0m 15.68s

 // Do you want to update wxt.theme key in wxt_library.settings config?: yes.   

 [success] Cache rebuild complete.

  1/13 [==>-------------------------]   7%
  6/13 [============>---------------]  46%
 10/13 [=====================>------]  76%
 13/13 [============================] 100% [notice] Processed 13 items (13 created, 0 updated, 0 failed, 0 ignored) - done with 'wxt_file'
 [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'wxt_file'

 1/2 [==============>-------------]  50%
 2/2 [============================] 100% [notice] Processed 2 items (2 created, 0 updated, 0 failed, 0 ignored) - done with 'wxt_node_page'

  1/13 [==>-------------------------]   7%
  3/13 [======>---------------------]  23%
  6/13 [============>---------------]  46%
 10/13 [=====================>------]  76%
 12/13 [=========================>--]  92%
 13/13 [============================] 100% [notice] Processed 13 items (13 created, 0 updated, 0 failed, 0 ignored) - done with 'wxt_media'

 1/3 [=========>------------------]  33%
 3/3 [============================] 100% [notice] Processed 3 items (3 created, 0 updated, 0 failed, 0 ignored) - done with 'wxt_media_slideshow'

 1/4 [=======>--------------------]  25%
 4/4 [============================] 100% [notice] Processed 4 items (4 created, 0 updated, 0 failed, 0 ignored) - done with 'gcweb_block'

 1/3 [=========>------------------]  33%
 3/3 [============================] 100% [notice] Processed 3 items (3 created, 0 updated, 0 failed, 0 ignored) - done with 'gcweb_block_spotlight'

 1/2 [==============>-------------]  50%
 2/2 [============================] 100% [notice] Processed 2 items (2 created, 0 updated, 0 failed, 0 ignored) - done with 'gcweb_node_landing_page'

  1/38 [>---------------------------]   2%
  8/38 [=====>----------------------]  21%
 16/38 [===========>----------------]  42%
 23/38 [================>-----------]  60%
 27/38 [===================>--------]  71%
 35/38 [=========================>--]  92%
 38/38 [============================] 100% [notice] Processed 38 items (38 created, 0 updated, 0 failed, 0 ignored) - done with 'gcweb_menu_link'

4) Port Forward

kubectl port-forward -n default svc/drupal-varnish 5000:80

bernardmaltais commented 4 years ago

@sylus That was quick! It does indeed deploy perfectly now! Nice work guys! We are debating internally between building our deployment in Azure App Services using containers or leveraging this great piece of work. Obviously I am for reusing your work as this is the reason you make this open and available with great support.

I have a few questions regarding how to best operate the solution once deployed so I will probably reach out eventually to discuss what you would consider best day two operations practices.

sylus commented 4 years ago

Awesome to hear and yeah Zachary is going to get back to your email soon we did a whole write up about some of the downsides of Azure Web Apps (and why is disabled at our dept since we have a platform team) and Zachary will give you a detailed reply. We could also share our security controls, and concept of operations that we recieved an ATO for our cloud platform with.

In essence Azure Web Apps is okay but you won't really get to much control or useful scaling. However if you are only doing one app then it might makes sense. For us we use our cloud native platform so logging in our Elastic stack using the Elastic on Cloud operator, credentials in Hashicorp Vault, and real time metrics in Prometheus / Grafana. Then having everything under Mutual TLS through the Istio service mesh. The value is only really achieved when you have multiple workloads running under it though due to economy of scale and making it an organizational shift / priority to adopt that workflow and invest in cloud native tooling. Another big reason is of course the community we get to work with and ability to jump to any cloud. Also that we are GCEARB endorsed.

https://github.com/canada-ca-terraform-modules/terraform-kubernetes-aks https://github.com/canada-ca-terraform-modules/terraform-kubernetes-aks-platform

bernardmaltais commented 4 years ago

Awesome! This will be very helpful for us. It is nice to stand on the shoulder of a giant ;-)

sylus commented 4 years ago

@bernardmaltais

If you get a chance try out Kind for local dev https://kind.sigs.k8s.io/

I tested varnish myself and worked and showed all the steps here:

https://github.com/drupalwxt/helm-drupal/issues/38#issuecomment-731815239

sylus commented 4 years ago

Closing this out thanks for testing it :D

sylus commented 4 years ago

@bernardmaltais

Remember we do have this here but it doesn't have the pgbouncer and varnish configurations:

https://github.com/drupalwxt/terraform-containers-webapp-azure

Also we have this if you decide to go for Managed Postgresql which our chart supports external dbs:

https://github.com/canada-ca-terraform-modules/terraform-azurerm-postgresql

sylus commented 4 years ago

@bernardmaltais just in case you decide to test the redis integration you will need to run chart version 0.6.6 and point to these images until the next release as we just added the redis library https://github.com/drupalwxt/docker-scaffold/commit/1deda6b47e65a94618801e2e807e7821a3a045d6

drupalwxt/site-wxt:4.0.0-rc1-redis drupalwxt/site-wxt:40.0-rc1-redis-nginx

sylus commented 4 years ago

@bernardmaltais sorry chart version 0.6.7+

drupalwxt / helm-drupal

varnish pod crashloopback #38