I was using your image to troubleshoot some configuration in the multi-container Carto repo I'm working on, and noticed that Varnish as configured wasn't actually receiving traffic. This PR has Nginx and Varnish config changes that set Varnish up to cache both the SQL API and Windshaft. It also explicitly forces Nginx not to cache those requests, which it tends to do by default (potentially creating two layers of caching, with lots of phantom HTTP 304 bugs as a result). There are more descriptive details in the commit messages, and feel free to @ me if anything is unclear.
A few helpful commands (to run on the container) for testing this out:
tail -f /var/log/nginx/cartodb_access.log
/opt/varnish/bin/varnishlog -c - shows Varnish logs for requests from the client
tail -f /CartoDB-SQL-API/logs/cartodb-sql-api.log
If you run those in separate sessions and then make a new request that hits the SQL API, you should be able to see the request get proxied by Nginx, passed to Varnish, then passed on to the SQL API itself. A subsequent request for the same resource should (barring things like using the Authorization header, which will cause Varnish to automatically pass the request) hit Nginx and Varnish, but not make it all the way to the SQL API.
Also, if you keep the varnish logging process open while making style changes to a visualization in the Builder interface, you will see PURGE requests being processed by Varnish, based on HTTP calls made by the invalidation logic in the postgres trigger created by the code here:
sverhoeven
commented
5 years ago
Hi!
I was using your image to troubleshoot some configuration in the multi-container Carto repo I'm working on, and noticed that Varnish as configured wasn't actually receiving traffic. This PR has Nginx and Varnish config changes that set Varnish up to cache both the SQL API and Windshaft. It also explicitly forces Nginx not to cache those requests, which it tends to do by default (potentially creating two layers of caching, with lots of phantom HTTP 304 bugs as a result). There are more descriptive details in the commit messages, and feel free to @ me if anything is unclear.
A few helpful commands (to run on the container) for testing this out:
tail -f /var/log/nginx/cartodb_access.log/opt/varnish/bin/varnishlog -c- shows Varnish logs for requests from the clienttail -f /CartoDB-SQL-API/logs/cartodb-sql-api.logIf you run those in separate sessions and then make a new request that hits the SQL API, you should be able to see the request get proxied by Nginx, passed to Varnish, then passed on to the SQL API itself. A subsequent request for the same resource should (barring things like using the
Authorizationheader, which will cause Varnish to automatically pass the request) hit Nginx and Varnish, but not make it all the way to the SQL API.Also, if you keep the varnish logging process open while making style changes to a visualization in the Builder interface, you will see PURGE requests being processed by Varnish, based on HTTP calls made by the invalidation logic in the postgres trigger created by the code here:
https://github.com/CartoDB/cartodb/blob/05a05fd6959bf4cc42480daec08d28449532cd8e/app/models/user/db_service.rb#L1491-L1538
Hope this is useful,
Nick