search

JavaMoney / jsr354-ri

JSR 354 - Moneta: Reference Implementation
Other
343 stars 101 forks source link

Run "ConversionExample", Got "java.io.IOException: Request has been rejected by IMF server." #191

Closed guofengzh closed 6 years ago

guofengzh commented 6 years ago

I run ConversionExample.java in javamoney-examples, JDK 1.8, moneta 1.2.1, I got the following WARNING about IMFProvider:

Failed to read/load resource (checking fallback): IMFRateProvider java.lang.IllegalArgumentException: Failed to load new data: org.javamoney.moneta.convert.imf.IMFRateProvider{ context: ProviderContext ( {User-Agent=Chrome/51.0.2704.103, rateTypes=[DEFERRED], providerDescription=International Monetary Fond, days=1, provider=IMF})} at org.javamoney.moneta.internal.loader.DefaultLoaderListener.trigger(DefaultLoaderListener.java:85) at org.javamoney.moneta.internal.loader.LoadDataLoaderService.execute(LoadDataLoaderService.java:41) at org.javamoney.moneta.internal.loader.DefaultLoaderServiceFacade.loadData(DefaultLoaderServiceFacade.java:43) at org.javamoney.moneta.internal.loader.DefaultLoaderService.loadData(DefaultLoaderService.java:264) at org.javamoney.moneta.convert.imf.IMFRateProvider.<init>(IMFRateProvider.java:62) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.javamoney.moneta.internal.PriorityAwareServiceProvider.loadServices(PriorityAwareServiceProvider.java:98) at org.javamoney.moneta.internal.PriorityAwareServiceProvider.getServices(PriorityAwareServiceProvider.java:65) at javax.money.spi.Bootstrap.getServices(Bootstrap.java:102) at org.javamoney.moneta.convert.internal.DefaultMonetaryConversionsSingletonSpi.reload(DefaultMonetaryConversionsSingletonSpi.java:64) at org.javamoney.moneta.convert.internal.DefaultMonetaryConversionsSingletonSpi.<init>(DefaultMonetaryConversionsSingletonSpi.java:56) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.javamoney.moneta.internal.PriorityAwareServiceProvider.loadServices(PriorityAwareServiceProvider.java:98) at org.javamoney.moneta.internal.PriorityAwareServiceProvider.getServices(PriorityAwareServiceProvider.java:65) at javax.money.spi.Bootstrap.getService(Bootstrap.java:113) at javax.money.convert.MonetaryConversions.getMonetaryConversionsSpi(MonetaryConversions.java:59) at javax.money.convert.MonetaryConversions.getConversion(MonetaryConversions.java:89) at javax.money.convert.MonetaryConversions.getConversion(MonetaryConversions.java:105) at org.javamoney.examples.console.simple.conversion.ConversionExample.main(ConversionExample.java:40) Caused by: java.lang.IllegalArgumentException: Failed to load IMF data provided. at org.javamoney.moneta.convert.imf.IMFAbstractRateProvider.newDataLoaded(IMFAbstractRateProvider.java:123) at org.javamoney.moneta.convert.imf.IMFRateProvider.newDataLoaded(IMFRateProvider.java:39) at org.javamoney.moneta.internal.loader.DefaultLoaderListener.trigger(DefaultLoaderListener.java:83) ... 32 more Caused by: java.io.IOException: Request has been rejected by IMF server. at org.javamoney.moneta.convert.imf.IMFRateReadingHandler.read(IMFRateReadingHandler.java:66) at org.javamoney.moneta.convert.imf.IMFAbstractRateProvider.newDataLoaded(IMFAbstractRateProvider.java:114) ... 34 more

and

Failed to read/load resource (checking fallback): IMFHistoricRateProvider java.lang.IllegalArgumentException: Failed to load new data: org.javamoney.moneta.convert.imf.IMFHistoricRateProvider{ context: ProviderContext ( {User-Agent=Chrome/51.0.2704.103, rateTypes=[HISTORIC], providerDescription=Historic International Monetary Fond, days=0, provider=IMF-HIST})} at org.javamoney.moneta.internal.loader.DefaultLoaderListener.trigger(DefaultLoaderListener.java:85) at org.javamoney.moneta.internal.loader.LoadDataLoaderService.execute(LoadDataLoaderService.java:41) at org.javamoney.moneta.internal.loader.DefaultLoaderServiceFacade.loadData(DefaultLoaderServiceFacade.java:43) at org.javamoney.moneta.internal.loader.DefaultLoaderService.loadData(DefaultLoaderService.java:264) at org.javamoney.moneta.convert.imf.IMFHistoricRateProvider.<init>(IMFHistoricRateProvider.java:66) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.javamoney.moneta.internal.PriorityAwareServiceProvider.loadServices(PriorityAwareServiceProvider.java:98) at org.javamoney.moneta.internal.PriorityAwareServiceProvider.getServices(PriorityAwareServiceProvider.java:65) at javax.money.spi.Bootstrap.getServices(Bootstrap.java:102) at org.javamoney.moneta.convert.internal.DefaultMonetaryConversionsSingletonSpi.reload(DefaultMonetaryConversionsSingletonSpi.java:64) at org.javamoney.moneta.convert.internal.DefaultMonetaryConversionsSingletonSpi.<init>(DefaultMonetaryConversionsSingletonSpi.java:56) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.javamoney.moneta.internal.PriorityAwareServiceProvider.loadServices(PriorityAwareServiceProvider.java:98) at org.javamoney.moneta.internal.PriorityAwareServiceProvider.getServices(PriorityAwareServiceProvider.java:65) at javax.money.spi.Bootstrap.getService(Bootstrap.java:113) at javax.money.convert.MonetaryConversions.getMonetaryConversionsSpi(MonetaryConversions.java:59) at javax.money.convert.MonetaryConversions.getConversion(MonetaryConversions.java:89) at javax.money.convert.MonetaryConversions.getConversion(MonetaryConversions.java:105) at org.javamoney.examples.console.simple.conversion.ConversionExample.main(ConversionExample.java:40) Caused by: java.lang.IllegalArgumentException: Failed to load IMF data provided. at org.javamoney.moneta.convert.imf.IMFAbstractRateProvider.newDataLoaded(IMFAbstractRateProvider.java:123) at org.javamoney.moneta.convert.imf.IMFHistoricRateProvider.newDataLoaded(IMFHistoricRateProvider.java:47) at org.javamoney.moneta.internal.loader.DefaultLoaderListener.trigger(DefaultLoaderListener.java:83) ... 32 more Caused by: java.io.IOException: Request has been rejected by IMF server. at org.javamoney.moneta.convert.imf.IMFRateReadingHandler.read(IMFRateReadingHandler.java:66) at org.javamoney.moneta.convert.imf.IMFAbstractRateProvider.newDataLoaded(IMFAbstractRateProvider.java:114) ... 34 more

How to deal with this issue?

keilw commented 6 years ago

Hard to say, do you try this from a "strange" country like Russia, Turkey, etc. where there have been several restrictions or hacker attacks in recent weeks that may cause the IMF to ban certain countries or IP ranges?

guofengzh commented 6 years ago

This example is run on my home desktop in Beijing, China. But I use javamoney 1.2.1 in a project run on AWS in Korea. It has the same issue. Loading the data from ECB works well. I understand that IMFProvider works but has limited access in some countries. Thanks for your support.

keilw commented 6 years ago

This is thrown on a more regular basis. And mostly affects the "Last Five Days" like http://www.imf.org/external/np/fin/data/rms_five.aspx While other endpoints including Current Month have no problems.

Right now RMS_FIVE is the default, it is worth looking at alternative strategies, e.g. use historic/current month as a fallback.

keilw commented 6 years ago

It's quite bizarre because in the browser "Last Five Days" work again, but unit tests still fail for javamoney-examples. Maybe batch access or some attributes/user agent, etc. are not accepted any more?

McPringle commented 6 years ago

I can confirm that this problem is not IP or country specific. I tried it from several countries (Latvia, Germany, Switzerland, Unites States) with the same result starting with the first request:

Caused by: java.io.IOException: Request has been rejected by IMF server.
    at org.javamoney.moneta.convert.imf.IMFRateReadingHandler.read(IMFRateReadingHandler.java:66)
    at org.javamoney.moneta.convert.imf.IMFAbstractRateProvider.newDataLoaded(IMFAbstractRateProvider.java:114)
    ... 34 more

This is my demo application: https://github.com/McPringle/jsr-354-demo/blob/master/src/main/java/ch/fihlon/jsr354/demo/CurrencyConversionDemo.java

moneyApiVersion = 1.0.3 monetaVersion = 1.2.1

keilw commented 6 years ago

It seems to affect that 5 day query more than others. Not sure, what to do about it, but with a possible slight performance penalty we could try to access other endpoints/data sets that usually return a whole month.

c-delanneau commented 6 years ago

Hello,

I had the same problem and I managed to solve it as follows:

In the file "LoadableResource.java" ("org.javamoney.moneta.internal.loader"), you must specify the "User-Agent" in the case of an http(s) request.

To do this, in the protected method "load", I added the following code after line 325:

if (conn instanceof HttpURLConnection) {
    conn.setRequestProperty("User-Agent", "Chrome/51.0.2704.103");
}

Hoping that it can help you.

keilw commented 6 years ago

Thanks, would you be able to propose a PR with that?

c-delanneau commented 6 years ago

Yes, I'll do that tomorrow.

keilw commented 6 years ago

After merging the PR it still gets rejected:

java.lang.IllegalArgumentException: Failed to load new data: org.javamoney.moneta.convert.imf.IMFRateProvider{ context: ProviderContext (
{User-Agent=Chrome/51.0.2704.103, rateTypes=[DEFERRED], providerDescription=International Monetary Fond, days=1, provider=IMF})}
    at org.javamoney.moneta.internal.loader.DefaultLoaderListener.trigger(DefaultLoaderListener.java:85)
    at org.javamoney.moneta.internal.loader.LoadDataLoaderService.execute(LoadDataLoaderService.java:41)
    at org.javamoney.moneta.internal.loader.DefaultLoaderServiceFacade.loadData(DefaultLoaderServiceFacade.java:43)
    at org.javamoney.moneta.internal.loader.DefaultLoaderService.loadData(DefaultLoaderService.java:264)
    at org.javamoney.moneta.convert.imf.IMFRateProvider.<init>(IMFRateProvider.java:62)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
    at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:777)
    at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:721)
    at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1389)
    at org.javamoney.moneta.internal.PriorityAwareServiceProvider.loadServices(PriorityAwareServiceProvider.java:98)
    at org.javamoney.moneta.internal.PriorityAwareServiceProvider.getServices(PriorityAwareServiceProvider.java:65)
    at javax.money.spi.Bootstrap.getServices(Bootstrap.java:102)
    at org.javamoney.moneta.convert.internal.DefaultMonetaryConversionsSingletonSpi.reload(DefaultMonetaryConversionsSingletonSpi.java:64)
    at org.javamoney.moneta.convert.internal.DefaultMonetaryConversionsSingletonSpi.<init>(DefaultMonetaryConversionsSingletonSpi.java:56)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
    at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:777)
    at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:721)
    at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1389)
    at org.javamoney.moneta.internal.PriorityAwareServiceProvider.loadServices(PriorityAwareServiceProvider.java:98)
    at org.javamoney.moneta.internal.PriorityAwareServiceProvider.getServices(PriorityAwareServiceProvider.java:65)
    at javax.money.spi.Bootstrap.getService(Bootstrap.java:113)
    at javax.money.convert.MonetaryConversions.getMonetaryConversionsSpi(MonetaryConversions.java:59)
    at javax.money.convert.MonetaryConversions.getExchangeRateProvider(MonetaryConversions.java:169)
    at javax.money.convert.MonetaryConversions.getExchangeRateProvider(MonetaryConversions.java:197)
    at org.javamoney.moneta.convert.imf.IMFHistoricRateProviderTest.setup(IMFHistoricRateProviderTest.java:61)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:100)
    at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:515)
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:216)
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:143)
    at org.testng.TestRunner.beforeRun(TestRunner.java:624)
    at org.testng.TestRunner.run(TestRunner.java:592)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
    at org.testng.SuiteRunner.run(SuiteRunner.java:268)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1264)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1189)
    at org.testng.TestNG.runSuites(TestNG.java:1104)
    at org.testng.TestNG.run(TestNG.java:1076)
    at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:69)
    at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:181)
    at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99)
    at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:113)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: java.lang.IllegalArgumentException: Failed to load IMF data provided.
    at org.javamoney.moneta.convert.imf.IMFAbstractRateProvider.newDataLoaded(IMFAbstractRateProvider.java:123)
    at org.javamoney.moneta.convert.imf.IMFRateProvider.newDataLoaded(IMFRateProvider.java:39)
    at org.javamoney.moneta.internal.loader.DefaultLoaderListener.trigger(DefaultLoaderListener.java:83)
    ... 57 more
Caused by: java.io.IOException: Request has been rejected by IMF server.
    at org.javamoney.moneta.convert.imf.IMFRateReadingHandler.read(IMFRateReadingHandler.java:66)
    at org.javamoney.moneta.convert.imf.IMFAbstractRateProvider.newDataLoaded(IMFAbstractRateProvider.java:114)
    ... 59 more

One can see, the UA is applied, but it has no effect on the test run.

guofengzh commented 6 years ago

I tried @c-delanneau code and it works very well. But I use the following and it works as well too in our desktop and the product server on AWS in Korea:

if (conn instanceof HttpURLConnection) {
                conn.setRequestProperty("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.63 Safari/537.36");

            }
keilw commented 6 years ago

This sounds a little broader than just Chrome. Thanks @guofengzh. We may give it a try. Still strange, that wherever Travis-CI (the company itself is in Berlin) has its build servers it does not seem to have an effect, but if a production server on AWS in Korea works that is a good sign. Ideally it should for most countries or AWS regions. I could imagine that making it configurable somehow instead of hard-coding it was also helpful.

keilw commented 6 years ago

This was fixed with https://github.com/JavaMoney/jsr354-ri/pull/203

brassier commented 3 years ago

Is it possible that something like this is happening in other environments? We have a couple spring boot apps that are struggling to find the default "5 day" resource. This URL seems to work fine in a browser though.

Apr 26 18:03:08  2021-04-26 18:03:08.057  INFO 30680 --- [nio-8010-exec-2] o.j.m.internal.loader.LoadableResource   : Failed to load resource input for IMFHistoricRateProvider from https://www.imf.org/external/np/fin/data/rms_five.aspx?tsvflag=Y
Apr 26 18:03:08  java[30680]: javax.net.ssl.SSLException: Received fatal alert: internal_error
keilw commented 3 years ago

This is the first time in almost 3 years anybody mentions anything here, so it doesn't seem to frequent. Of course the IMF could tighten security to try avoid attacks like DDNS or similar. We saw in project Agorava that some APIs like Facebook, Twitter and others work fine from a browser or mobile but once you try to access the same within a Spring or Java EE/Jakarta EE like environment, it would reject those requests with some weird "possible attack" message.

Beside this was closed and resolved back then.

McPringle commented 3 years ago

Tipp: Check the user agent, your spring boot app uses. Try to make it look like a browser and don't use your application name. That solved a lot of problems in my apps where server-side protection kicked in on "suspicious" activity (like a "suspicious" user agent).

keilw commented 3 years ago

Thanks a lot, if you had time because at least occasionally Spring Boot or other web app users may face similar issues, you're welcome to contribute to a quick tip or similar either in a Wiki (https://github.com/JavaMoney/javamoney-examples/wiki among others, there is no Wiki here, and I'm not sure, if it is the best place but if you prefer this repo just say) or similar place? I also sent you an invitation @McPringle to join the organization as "Associate" if that's not enough to edit a Wiki we could also tweak the team, Contributor is normally for contributing JCP members, if you're already a JCP member, you'd be more than welcome to join that team even without listing on jcp.org.