archive_day function with yahoodayfetcher

[trbck]

Hi yinhm

I tried to add the yahoo provider right within the datafeed server loop of my application:

http://pastebin.com/XsBSzZsC

Every time it executes the loop and therefore the whole script terminates right after the yahoo provider has done the job. I can only assume this way to do it is not applicable to the tornado io loop of the datafeed server. Do you have any suggestions? I don't want to place my archive_day function in another script and upload it with client.put_day() anymore.

[yinhm]

Hi @trbck

This is due to http fetcher will stop the ioloop instance after all request processed. See: https://github.com/yinhm/datafeed/blob/master/datafeed/providers/http_fetcher.py#L79

When I design the fetcher it is mean to be using as a standalone scripts, so I simply made the choice to using a global ioloop for http client. That is the root cause.

This can be avoid by isolating the server ioloop and the http fetcher ioloop.

I will commit a change to the master shortly.

[trbck]

Hi, thanks for the update but now the main loop blocks until the fetcher is finished. Since I want to fetch a lot of symbols it takes a while for the datafeed server to respond to external requests. Do you know a way to fix this?

[yinhm]

Hi @trbck

The server didn't design around async, so the main loop blocks when the archive_day gets called.

I think it may a lot easier to using another scripts to sync the data.

[trbck]

yes, I integrated it with https://github.com/nvie/rq now. Thanks so much for your help!