IMF Rate Providers can be very slow to update

[brassier]

I notice some hanging when grabbing fresh data, specifically with the IMF resources. I see that there is an override for useragent in the properties for this provider, likely for similar issues.

I made some straight HTTP requests to one of the endpoints in question (https://www.imf.org/external/np/fin/data/rms_five.aspx?tsvflag=Y). In a browser it downloads quickly. Taking that as a default, and slowly stripping headers away I notice that some of the Accept headers seem to cause this request to be very slow when removed (8-10 seconds). This is compared to <1 second with these headers. In my testing, removing any one of the headers below causes this slowdown.

Accept: application/xhtml+xml
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

Is there any way we could get a similar override like the useragent override for these requests? Or is there a contact at the IMF vendor that could remove this limitation?

[keilw]

I don't think we ever had contact with them, their website has all sorts of campaigns about COVID support etc. but no real developer relationship or so. https://www.imf.org/en/Data looks like the part dealing with those datapoints, therefore please check them out. It is free to use so can't say if they offer any support or force people to pay for it or throttle the free APIs to avoid overloading them, that is possible. Users can alway create their own providers, the ones coming with the RI are a basic set, large companies like Zalando are known to have their own.

[brassier]

Thanks @keilw. I will reach out to them and see if it's anything they can help with on their side.

[Dudeplayz]

At every app start, I get spammed multiple times with:

LoadableResource - Failed to load resource input for IMFHistoricRateProvider from https://www.imf.org/external/np/fin/data/rms_five.aspx?tsvflag=Y
java.net.SocketTimeoutException: Read timed out

After some tries, it is loaded. Could be related to this issue.

[keilw]

@Dudeplayz Yes that sounds like it.

[brassier]

@Dudeplayz - you may be able to leverage something like below in your javamoney.properties file to avoid the read timeouts. The default is 10 Seconds which often is not long enough for this 3rd party.

{1}load.IMFRateProvider.connection.read.timeout=20
{1}load.IMFRateProvider.connection.connect.timeout=20

[Dudeplayz]

@brassier I tried it, but it doesn't work. I am also not able to disable it. If I remove the IMF dependency from maven I always get Invalid ExchangeRateProvider (not found): IMF.

[keilw]

Which Moneta version are each of you using?

[Dudeplayz]

1.4.2

[keilw]

Neither the timeouts nor the UserAgent or headers look like they're working anymore in a console/Maven build.

[keilw]

• [x] Use curl instead of wget in the shell script for now
• [x] Different structure of the default file to overcome JPMS loading problem
• [x] Try to solve the timeout by either using
  • Java HttpClient (would require Java 11, probably not until 1.5)
  • Apache HttpClient
  • OkHttp
  • Retrofit
  • JCurl
  • Java Curl

Here are further options/examples including some above: https://www.mastertheboss.com/java/top-solutions-for-java-http-clients/?utm_content=cmp-true

[keilw]

Based on above checklist, let's look into a comparison of Java HTTP clients. The inner workings of IMFRemoteSearchCallable have to be updated to use something other than URLConnection which works better for the IMF servers, but LoadableURLResource in "core" also has to be rewritten

[keilw]

Fixed with #400

[keilw]

@brassier This was fixed with 1.4.3 The actual problem was accessing the IMF and other exchange rates using the old JDK URLConnection. This was fixed by using Okhttp3. Where passing headers like

Accept: application/xhtml+xml
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

or the Useragent, which was actually found counterproductive because it caused errors like java.io.IOException: stream was reset. Hence the string is ignored and will likely be deprecated/removed in a future version.