Closed koush closed 3 years ago
Oh, saw the same error in our logs, didn't know where that came from. @koush did you find a way to fix this?
As a long running service, my deployment was unreliable. After some time, the service would drop to serving assets at 100Kbps (on a 10Gbps connection). Wait long enough, it would become completely unresponsive. I eventually traced it down to the Github proxying and/or asset caching. Something was leaking, and I think this error was a manifestation of the leak. I couldn't find a fix for that, and wasn't that inclined to due to my infrastructure.
Since I am behind Nginx anyways, I added some code to offload caching to that. So my nuts deployment serves an asset (no caching), and that response gets cached by nginx for 1 hour. After that, all subsequent downloads are entirely handled by nginx.
I don't think nuts should be reinventing http caching.
Here's my fix:
commit 664e6661da22d992b668eaaa46112b905bc05b5f
Author: Koushik Dutta <koushd@gmail.com>
Date: Sat May 6 09:42:01 2017 -0700
add cache directives for being behind a reverse proxy
diff --git a/bin/web.js b/bin/web.js
index 9e7c0aa..27807da 100644
--- a/bin/web.js
+++ b/bin/web.js
@@ -26,7 +26,8 @@ var myNuts = nuts.Nuts({
timeout: process.env.VERSIONS_TIMEOUT,
cache: process.env.VERSIONS_CACHE,
refreshSecret: process.env.GITHUB_SECRET,
- proxyAssets: !Boolean(process.env.DONT_PROXY_ASSETS)
+ proxyAssets: !Boolean(process.env.DONT_PROXY_ASSETS),
+ behindReverseProxy: Boolean(process.env.BEHIND_REVERSE_PROXY),
});
// Control access to API
diff --git a/env.sh b/env.sh
index 6be1959..6c87244 100644
--- a/env.sh
+++ b/env.sh
@@ -17,3 +17,4 @@ export GITHUB_REPO=koush/vysor.io
# behind a reverse proxy like nginx
# http://expressjs.com/en/guide/behind-proxies.html
export TRUST_PROXY=loopback
+export BEHIND_REVERSE_PROXY=true
diff --git a/lib/backends/backend.js b/lib/backends/backend.js
index eec766a..db936a6 100644
--- a/lib/backends/backend.js
+++ b/lib/backends/backend.js
@@ -75,6 +75,15 @@ Backend.prototype.serveAsset = function(asset, req, res) {
return that.getAssetStream(asset)
.then(function(stream) {
+ if (that.opts.behindReverseProxy) {
+ return Q(stream
+ .on('response', function(res) {
+ delete res.headers['etag'];
+ res.headers['cache-control'] = 'max-age=3600'
+ })
+ .pipe(res));
+ }
+
return Q.all([
// Cache the stream
that.cache.set(cacheKey, stream),
diff --git a/lib/backends/github.js b/lib/backends/github.js
index 3eaf155..21f46cd 100644
--- a/lib/backends/github.js
+++ b/lib/backends/github.js
@@ -88,5 +88,37 @@ GitHubBackend.prototype.getAssetStream = function(asset) {
}));
};
+GitHubBackend.prototype.readAsset = function(asset) {
+ var d = Q.defer();
+ var headers = {
+ 'User-Agent': 'nuts',
+ 'Accept': 'application/octet-stream'
+ };
+ var httpAuth;
+
+ if (this.opts.token) {
+ headers['Authorization'] = 'token '+this.opts.token;
+ } else if (this.opts.username) {
+ httpAuth = {
+ user: this.opts.username,
+ pass: this.opts.password,
+ sendImmediately: true
+ };
+ }
+
+ request({
+ uri: asset.raw.url,
+ method: 'get',
+ headers: headers,
+ auth: httpAuth
+ }, function(err, message, data) {
+ if (err) {
+ return d.reject(err);
+ }
+ return d.resolve(data);
+ })
+
+ return d.promise;
+};
module.exports = GitHubBackend;
Thanks @koush. Not sure how I'll proceed though.
I ended up setting env DONT_PROXY_ASSETS=true
, because I'm hosting my releases on a public Github repo. Will see if this solves the issue for me.
Here's a couple the last bit of stuff in the logs: