search

spacecodewor / fmpcloud-go

Go API client for Financial Modeling Prep (fmpcloud.io)
MIT License
24 stars 14 forks source link

Pulling Data Results In "No Host" Error Eventually #7

Open FrankDMartinez opened 3 years ago

FrankDMartinez commented 3 years ago

Hi, I hope you are having a great new year so far. Thanks for writing and maintaining this package.

In our application, we pull all the available tickers, sort them alphabetically, and then poll the API for balance-sheet information for each, currently in groups of anywhere between 2 and 5 concurrently with pauses in between to stay within the 10/second limit (we were initially planning on groups of 10 concurrently separated by 1 second pauses when we noticed this behavior). The "no such host" errors always seem to arise with the area of the same group, curiously enough, roughly 40-45 seconds after the start of the balance-sheet data pull. So, I don't know if that is a factor.

For example, I just tried pulling the data in groups of 5 with a 0.5 second delay between each group and received the error for the group with tickers 0272.HK, 0273.HK, 0274.HK, 0276.HK, and 0277.HK. If I run it again, the error appears in the same place.

I have reached out to FMP and they are stumped. Any ideas as to why this behaviour occurs? Any information I can provide to help prompt a solution?

Thanks in advance.

FrankDMartinez commented 3 years ago

I think this might be a case of running into max open file descriptors; does the package close them?

FrankDMartinez commented 3 years ago

Taking a look at api.go and company_valuation.go, I am presuming the answer to my most recent question is "no". Would the solution then be to call .Body.Close() on the response after unmarshalling in company_valuation.go and before returning in api.go?

spacecodewor commented 3 years ago

Hi, I hope you are having a great new year so far. Thanks for writing and maintaining this package.

In our application, we pull all the available tickers, sort them alphabetically, and then poll the API for balance-sheet information for each, currently in groups of anywhere between 2 and 5 concurrently with pauses in between to stay within the 10/second limit (we were initially planning on groups of 10 concurrently separated by 1 second pauses when we noticed this behavior). The "no such host" errors always seem to arise with the area of the same group, curiously enough, roughly 40-45 seconds after the start of the balance-sheet data pull. So, I don't know if that is a factor.

For example, I just tried pulling the data in groups of 5 with a 0.5 second delay between each group and received the error for the group with tickers 0272.HK, 0273.HK, 0274.HK, 0276.HK, and 0277.HK. If I run it again, the error appears in the same place.

I have reached out to FMP and they are stumped. Any ideas as to why this behaviour occurs? Any information I can provide to help prompt a solution?

Thanks in advance.

Hello, after completing a request for your tickers, I do not receive an error. It would be great to see a code example, a sequence of actions, so that I can reproduce this scenario on my own. I have not encountered this error yet. The API has the ability to make repeated requests, doesn't that help?

spacecodewor commented 3 years ago

Taking a look at api.go and company_valuation.go, I am presuming the answer to my most recent question is "no". Would the solution then be to call .Body.Close() on the response after unmarshalling in company_valuation.go and before returning in api.go?

In this case data.Body () returns [] bytes and does not have a .Close () method

But if you use data.RawResponse.Body () and get io.ReadCloser then you can of course use .Close ()

You can fork and test this theory yourself if you have all the data and then do MR. Or provide me with an example of how you are solving the problems and I will try to reproduce and fix it.

FrankDMartinez commented 3 years ago

One step ahead of you there. ;-) I forked the repository last night and will try to come up with an answer.

By the way, what's reasoning behind using resty instead of the standard net/http package?

spacecodewor commented 3 years ago

One step ahead of you there. ;-) I forked the repository last night and will try to come up with an answer.

By the way, what's reasoning behind using resty instead of the standard net/http package?

I once found this package, it has a lot of ready-made functions, and since then I have been using it, sometimes without it, if I need my own implementation.

I can think of doing this on a pure net/http client and then everyone gets to choose what to use.

But it takes time =)