Cached requests

[ikjahaa]

I'm currently in development of an application which uses your API to generate profits. At the moment there is a problem in which my profit has a bit of a delay, I think this is due to some cache problems ?

Does your api cache requests ?

I was looking into the code, which uses com.sun.jersey.api.client.config Would it be possible to define a cache header property ? cache-control: max-age=0, no-cache, no-store

Sorry i'm not familiar with this lib.

Regards

[ikjahaa]

Currently reading : http://stackoverflow.com/questions/10934316/jersey-default-cache-control-to-no-cache

[Nithanim]

I have never looked into how my api would react to time-critical data since I have never used it for such things. Actually, I have never used it myself at all. I only made sure with some examples that the returned objects retain the same information that the official api returns as json. All requests to my api are passed on directly to jersey without caching and the result from it are parsed and return to the caller. Unfortunately, this is also the first time I am using jersey so I have no idea how it works under the hood. When searching for information you have to watch out if the resource you found talks about the server or the client version (which is often not stated clearly but it seems easily distinguishable by the use of annotations, especially when you see something like @Path, @RequestMapping or similar which indicates the binding to a path to serve requests on a server.

These are the few resources I found:

• http://stackoverflow.com/questions/32052024/caching-in-jersey-client-based-on-last-modified-http-header
• http://stackoverflow.com/questions/15087831/how-to-cache-response-with-jersey
• http://venkateshmpv.blogspot.co.at/2013/09/http-client-caching-for-rest-services.html

where nothing is mentioned about caching of jersey directly. Since they ask or provide a workaround for caching I assume that it is not implemented and you have to do it yourself if you need it. However, I would still not rule cashing out because there is still the http api underneath that could potentially do it but I do not think that it does personally. EDIT: Which might be the problem if this exact http lib is used (and I think it is used) http://stackoverflow.com/questions/34534945/java-httpurlconnection-returns-cached-response-every-time

[ikjahaa]

Let me add abit info... My application makes a snapshot of your material storage, and another one when you're done. Then it compares these two for a profit result.

I discovered the issue when my second snapshot isnt always matching the users current material storage.... after some time ( might be the cache lifespan ) it catches up.

[Nithanim]

That sounds really bad if it even breaks here. But the problem lies deep in there and I am not sure if I could do something about it (actually falls in a sun package on my system so there is no guarantee on others). I tried redoing it so jersey uses the apache http client for the the past hours but I give up now (and probably for the near future). The three problems (jersey with apache http and that with the stupid ssl certificate) is too much for me to handle for now.

I have redone the config to support the default and the apache http client but for apache I do not know how to feed it the ssl cert because on the one hand I could use the PoolingHttpClientConnectionManager could easily use the ssl cert but this class is rejected by jersey-apache-client4 lib that attaches the apache http client to jersey because it expects the deprecated PoolingClientConnectionManager where I have yet to find something about ssl.

[Nithanim]

A little update on that matter. For the new http engine I started the migration from jersey 1 to 2 (which was about time and which I should have used from the beginning). I think am pretty much done with it although there are still some parts missing but gained a lot of insight doing that. You might have been on track from the beginning but I went on my own path too soon it seems - sorry about that. I still do not know the real cause of your problem for sure but I am pretty certain that it will be fixed after I am done. If it did not work for some reason I can make adjustments more easily at least.

[ikjahaa]

Sounds great !

Could you reproduce you issue though ? Just use your api to check your wallet's gold valuta. Then sell some stuff ingame to change the value. Check if your api has updated the value or still has the old gold value...

[Nithanim]

No, I have not tried to reproduce it with the game because I would have to pretty much download the game from scratch because have not played for a very long time. What I did was digging through sourcecode, reading about caching in http and looked how a real browser behaves when interacting with the json api.

Since I currently changed so much I cannot be sure that I did not break something (I am not even sure that the last snapshot didn't to be honest) so I am going to update the version number to not mess up other peoples builds. The changes for you will be on 0.0.6-SNAPSHOT when everything goes according to plan. This will be the first time that I am not using the maven-release-plugin but the artifactory plugin for jenkins from jfrog so I hope that I do not run into a lot of issues.

[Nithanim]

Had some initial problems (as always) but it seems that everything worked (except that there are no checksum files for release 0.0.5 in jcenter -.- I hope that maven still works without them). The snapshot should now be available as 0.0.6-SNAPSHOT in the snapshot repository (as listed in the readme). The result might be cached so you might want to use mvn install -U to force an update if you are using maven.

It would be nice if the combination of all changes fixes the issue for you. I had to change the config of the api so you have to use something like

GuildWars2Api gw2api = new GuildWars2Api(new GuildWars2ApiDefaultConfig().useGoDaddyFix(true).useApacheHttpClient(true));

now where you can test if the built in http and the apache http client return a different result for you.

edit: f39dbea28bb6507b3b8cea7d17e4988ac6c616ea

[ikjahaa]

I'll try this asap and get back to you with the result. Is the cache config set to no-cache by default, or do i set it manually somehow ?

[Nithanim]

Take your time! I do not know how much time I am going to have in the next weeks/months.

I assumed that if you make a request that you want to have the most up-to-date data (-> no caching).

I could have built in caching but I think that it is not really the responsibility of this api to do that. If I added that functionality it would require a lot more methods because cached data might be good most of the time but then you would need some parameter that disables it. I extracted the interface of this api already but I still have to link the current version against it and convert that to a maven module build. That would enable building a (caching) middle layer if it is really needed.

[ikjahaa]

I assumed that if you make a request that you want to have the most up-to-date data (-> no caching).

Yes, so would I. but just to clearify... your api doesn't cache by default configuration now ? (or at least shouldn't)

Then I'll test it later tonight...

[Nithanim]

Exactly, no caching is done. Every request is (should be) directly sent to the server and the most up-to-date result is returned.

What I have done in detail is that (1) I have updated to Jersey2 which enables the usage of the apache http client which either does not cache at all or is configurable (I am still not sure what the jersey-apachehttp-bridge uses though but I bet is changeable). (As a sidenote: Testing is also a lot easier now.) Additionally, (2) I added part of your request header which allows returning cached pages by intermediate servers but only if they are up-to-date. I do not know how or if such optimization works but I thought I would give it a try. I could always retry using no-cache it it is still a problem.

I hope that it works for you now!

[ikjahaa]

I would prefer that the default uses no-store. no-cache vs no-store

Also, the option to set these headers at runtime would be epic ! 😄 Cause for some instances I would not mind and maybe even prefer to use a cache... But for now a default no-store header should suffice for me.

Regards

[ikjahaa]

Still caching, could you add the no-store header ?

[Nithanim]

What a bummer that it did not work. However, before using the the last resort I would like to try no-cache as alternative still giving proxies a chance to return a cached result if the source did not change.

Since I can't think of a way to make headers configurable (especially per resource) in the short term, would it be a big issue for you if you used

try {
            Class<?> rh = gw2api.getClass().getClassLoader().loadClass("me.nithanim.gw2api.v2.util.rest.RequestHelperImpl");
            Field f = rh.getDeclaredField("DEFAULT_HEADERS");
            f.setAccessible(true);
            @SuppressWarnings("unchecked")
            MultivaluedHashMap<String, Object> headers = (MultivaluedHashMap<String, Object>) f.get(null); //all headers that are sent

            headers.putSingle("Cache-Control", "no-store"); 
        } catch(Throwable t) {
            System.out.println(t);
        }

to setup the default headers you want to use? You would have to do this only once at the beginning.

[ikjahaa]

Ill try your code tomorrow but,

I prefer preventing the ex occurance over try-catching... Wouldn't it be possible to having a public static header variable, Set it to default at startup. If i want to modify it, I can just access it and apply the modification so all future requests will be using my own defined headers.

just an idea though...

regards.

[ikjahaa]

Seems no-cache is sufficiant enough api changelog.

Though some endpoints have a server-side cache of 5 minutes which we cannot change... 5 minutes cache interval