Generic HTTP Client Pulling with HTTP Conditional Requests on GitHub API

[marcellodesales]

Hi there,

Problem

Hi guys,

I was trying to build the Node.js Client with pulling capabilities against out GitHub server. According to https://developer.github.com/guides/getting-started/#conditional-requests, we can control content caching by using the ETag header value from contents.

Question

How does the SpringCloud-Config clients make requests to GitHub Enterprise? Do you guys use this feature to identify when a resource has changed? Do you use the ETag and/or the Last-Modified HTTP Response Headers values???

Online API works

Here's the example of calling the API online with the proper 304 response.

First Request

I can see the

mdesales@Marcello-New2015 [09/19/2015 8:18:52] ~ $ 
curl -i https://api.github.com/users/marcellodesales

HTTP/1.1 200 OK
Server: GitHub.com
Date: Mon, 21 Sep 2015 21:57:41 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 1393
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 56
X-RateLimit-Reset: 1442875150
Cache-Control: public, max-age=60, s-maxage=60
Last-Modified: Thu, 17 Sep 2015 22:23:50 GMT
ETag: "36be0c1305782b56086b419619d90a4f"
Vary: Accept
X-GitHub-Media-Type: github.v3
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src 'none'
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
X-GitHub-Request-Id: C7108C1B:C026:C47BCD3:56007D55
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
Vary: Accept-Encoding
X-Served-By: 8dd185e423974a7e13abbbe6e060031e

{
  "login": "marcellodesales",
  "id": 131457,
  "avatar_url": "https://avatars.githubusercontent.com/u/131457?v=3",
  "gravatar_id": "",
  "url": "https://api.github.com/users/marcellodesales",
  "html_url": "https://github.com/marcellodesales",
  "followers_url": "https://api.github.com/users/marcellodesales/followers",
  "following_url": "https://api.github.com/users/marcellodesales/following{/other_user}",
  "gists_url": "https://api.github.com/users/marcellodesales/gists{/gist_id}",
  "starred_url": "https://api.github.com/users/marcellodesales/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/marcellodesales/subscriptions",
  "organizations_url": "https://api.github.com/users/marcellodesales/orgs",
  "repos_url": "https://api.github.com/users/marcellodesales/repos",
  "events_url": "https://api.github.com/users/marcellodesales/events{/privacy}",
  "received_events_url": "https://api.github.com/users/marcellodesales/received_events",
  "type": "User",
  "site_admin": false,
  "name": "Marcello de Sales",
  "company": "Intuit",
  "blog": "http://about.me/marcellodesales",
  "location": "San Diego, CA",
  "email": "marcello.desales@gmail.com",
  "hireable": null,
  "bio": null,
  "public_repos": 131,
  "public_gists": 19,
  "followers": 14,
  "following": 20,
  "created_at": "2009-09-26T00:38:09Z",
  "updated_at": "2015-09-17T22:23:50Z"
}

Second Request

Note that the ETag value did not change.

$ curl -i https://api.github.com/users/marcellodesales
HTTP/1.1 200 OK
Server: GitHub.com
Date: Mon, 21 Sep 2015 21:58:19 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 1393
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 55
X-RateLimit-Reset: 1442875150
Cache-Control: public, max-age=60, s-maxage=60
Last-Modified: Thu, 17 Sep 2015 22:23:50 GMT
ETag: "36be0c1305782b56086b419619d90a4f"
Vary: Accept
X-GitHub-Media-Type: github.v3
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src 'none'
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
X-GitHub-Request-Id: C7108C1B:ED5F:C538C71:56007D7B
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
Vary: Accept-Encoding
X-Served-By: 07ff1c8a09e44b62e277fae50a1b1dc4

{
  "login": "marcellodesales",
  "id": 131457,
  "avatar_url": "https://avatars.githubusercontent.com/u/131457?v=3",
  "gravatar_id": "",
  "url": "https://api.github.com/users/marcellodesales",
  "html_url": "https://github.com/marcellodesales",
  "followers_url": "https://api.github.com/users/marcellodesales/followers",
  "following_url": "https://api.github.com/users/marcellodesales/following{/other_user}",
  "gists_url": "https://api.github.com/users/marcellodesales/gists{/gist_id}",
  "starred_url": "https://api.github.com/users/marcellodesales/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/marcellodesales/subscriptions",
  "organizations_url": "https://api.github.com/users/marcellodesales/orgs",
  "repos_url": "https://api.github.com/users/marcellodesales/repos",
  "events_url": "https://api.github.com/users/marcellodesales/events{/privacy}",
  "received_events_url": "https://api.github.com/users/marcellodesales/received_events",
  "type": "User",
  "site_admin": false,
  "name": "Marcello de Sales",
  "company": "Intuit",
  "blog": "http://about.me/marcellodesales",
  "location": "San Diego, CA",
  "email": "marcello.desales@gmail.com",
  "hireable": null,
  "bio": null,
  "public_repos": 131,
  "public_gists": 19,
  "followers": 14,
  "following": 20,
  "created_at": "2009-09-26T00:38:09Z",
  "updated_at": "2015-09-17T22:23:50Z"
}

Third Request

Using "If-None-Match" value. It returned 304 as expected.

mdesales@Marcello-New2015 [09/19/2015 8:19:48] ~ $ 
$ curl -i -H 'If-None-Match: "36be0c1305782b56086b419619d90a4f"' https://api.github.com/users/marcellodesales

HTTP/1.1 304 Not Modified
Server: GitHub.com
Date: Mon, 21 Sep 2015 21:59:38 GMT
Status: 304 Not Modified
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 55
X-RateLimit-Reset: 1442875150
Cache-Control: public, max-age=60, s-maxage=60
Last-Modified: Thu, 17 Sep 2015 22:23:50 GMT
ETag: "36be0c1305782b56086b419619d90a4f"
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src 'none'
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
X-GitHub-Request-Id: C7108C1B:19B11:AB55AE4:56007DCA
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
Vary: Accept-Encoding
X-Served-By: b0ef53392caa42315c6206737946d931

Thanks for any help Marcello

[spencergibb]

Without knowing much about your problem, we don't do anything specific for github. We use jgit for all git communication.

[marcellodesales]

Hi @spencergibb

Thanks for your response... Really appreciated...

Without knowing much about Spring cloud Config, I'd like to know how your APIs know that a certain resource has changed... I understand that jgit is just the client for the content... In other words, are there flags that jgit gives you when a given content has changed... Can you point me to the specifics of your client so that I can investigate?

Our current GitHub Enterprise is not providing the proper ETag output and so there's no way to identify when a given resource changed... I'd like to investigate with you how to properly proceed in case I'm writing an HTTP Client with other language.

thanks a lot Marcello

[spencergibb]

In short it doesn't know when a resource has changed.

[marcellodesales]

@spencergibb Does it mean Services would be pulling the configuration every time not matter what??? Considering we are living in a Micro-services space, that would mean the configuration is returned at every pulling request... Is that correct?

Considering the HTTP Response Headers ETag or Last-Modified are provided, shouldn't those be used?

Async

I'm planning to use Message Queues (AMQP or AWS SNS) to support Async mode of the services. That way, I won't be pulling changes, but listening to the queue topic so that I can asynchronously give services (Java or Node.js) and indication that the Configuration has changed for a given environment.

[spencergibb]

How would I use ETag over git ssh?

[marcellodesales]

Hummm... Good point... I did not verify that jgit uses SSH...

I would think we could add a layer thought it would be pulling metadata through the GitHub API

https://developer.github.com/v3/repos/contents/#get-contents

Then, I would use jgit as it is to pull over ssh if the content has changed... :( I guess that's an extra layer of complexity... But at least it would properly pull whenever the content changes.

Options

I would also think a Git Hook could be used to callback the service to propagate the change to the clients.

[spencergibb]

Like I said, we need to be compatible with git, not github. We already have an open issue about pushing notifications to clients #86.

[marcellodesales]

@spencergibb Would it be possible to write a connector for GitHub instead of just Git??? We could take advantage of Conditional Requests.

[spencergibb]

yes, and webhooks.

[spencergibb]

This is fixed by 8d8e07a78e73c3e1e08f4277a76c6e1a4104f103

[marcellodesales]

@spencergibb... But I don't think that conditional requests are the same as webhooks... But I would think this is a workaround... thanks!