search

Rod-Persky / investoscope-easy-update

Updates Investoscope 3 end of day quotes
GNU General Public License v3.0
6 stars 3 forks source link

Process tags in [source]:[asset] format where source is MUTF_CA #4

Open bradsteels opened 6 years ago

bradsteels commented 6 years ago

Getting the following error messages (for mutual fund files). It seems to update ETFs (XIC.TO) No clue how to rectify. Is there an easy way to get Google finance data instead?

Unable to get information for TD Canadian Bond Index - e (MUTF_CA:TDB909) Unable to get information for Mawer New Canada (MUTF_CA:MAW107) Unable to get information for Chou RRSP (MUTF_CA:CHO102) Unable to get information for Chou Asia (MUTF_CA:CHO300) Unable to get information for CIBC Canadian Bond Index (MUTF_CA:CIB503) Traceback (most recent call last): File "investoscope_update.py", line 203, in main() File "investoscope_update.py", line 192, in main csv_text = get_quote(item) File "investoscope_update.py", line 140, in get_quote return get_yahoo_quote(item) File "investoscope_update.py", line 134, in get_yahoo_quote return download_yahoo_csv(item, yahoo_crumb, yahoo_cookie) File "investoscope_update.py", line 117, in download_yahoo_csv return tidy_yahoo_csv(csv_data.decode('utf-8')) File "investoscope_update.py", line 93, in tidy_yahoo_csv row.pop(5) IndexError: pop index out of range

Rod-Persky commented 6 years ago

At present I've only scraped data from the Yahoo Finance site. However, when you added the symbol MUTF_CA:CIB503 you've found a bug due to the assumption that the stock codes exists 👍

@bradsteels I'll look into how to extract data from the google finance pages, although it will be some time before I'll have time to do so.

charlietomo commented 6 years ago

I had the same issue IndexError: pop index out of range. However the script was NOT detailing which tickers it was having an issue with. I suspect your error is not related to the 5 tickers you have listed - the script isn't getting data for them, but it should complete otherwise. The error is with another ticker.

I traced it down to the underlying data on Yahoo - and some instruments do not have the same data as shares / the same data the script is looking for. If you look at VOD.L (https://finance.yahoo.com/quote/VOD.L/history?p=VOD.L) as an example of a share that works. If you go through all your shares that are set to Yahoo as a quote feed, if any don't look like the above link then change them to a manual feed, run the script again and see how you get on.

I went through 150+ pages, removed approx 10, then my error message went away - phew!

@Rod-Persky it would be great if the script could output which instrument it is having an error on - would save a lot of manual effort :-) Thanks for your work with this script.

Rod-Persky commented 6 years ago

I know this bug, but it isn't really to do with the yahoo data source 👍.

The reason it occurs is due to the assumption that the user added a ticker to their library using the search function, where the user selects yahoo as the data source and therefore the code should be formatted in the way yahoo expects. MUTF_CA:CIB503 is not a yahoo code (it's a google code) however it's likely that the user changed the quote source to yahoo. This app expects that the user uses the search function and only therefore the tickers are properly formatted yahoo formatted codes.

There are two options really:

1) Ignore the error, issue a warning "error occurred when accessing data" and move right along (user friendly) 2) Fix the scraper to ensure that the ticker code is correctly formatted and keep the warning (as it's not actually a bug)

I'll take action 1 for now and then look into fixing the script. The error probably will not help much.

bradsteels commented 6 years ago

Thanks for the update! Meanwhile the script is updating my ETFs that are listed in Yahoo and I'm certainly grateful for the time it saves.

Rod-Persky commented 6 years ago

@bradsteels I should still do something regarding this 👍

charlietomo commented 6 years ago

Regarding this it would be great if the script listed out the instrument that was having issues - rather than just failing cryptically "IndexError: pop index out of range".

Rod-Persky commented 6 years ago

Hi @charlietomo,

I should check the status of the scripts, but I think I did update the program to give better feedback last time you mentioned this 😎.

  1. Were you able to update since then?
  2. Would it be a good idea to add a mechanism to check for the latest release
  3. Should i simply tag some pushes as a release

Cheers!

charlietomo commented 6 years ago

Hi @Rod-Persky actually I've just upgraded to the app and it worked well - and I can't replicate my previous errors, so that is great.

  1. Yes I have updated.
  2. A mechanism to check for release would be good, if you are planning on regular updates :-)
  3. I'm not a big Github user so am not sure on the repercussions of this, but being clear on releases would be helpful - and having release notes. There is a massive jump (for the better) e.g. between script and app.
Rod-Persky commented 6 years ago

+1 release notes and some update mechanism would be fantastic additions. I will look into how this may be implemented in the future.

Rod-Persky commented 5 years ago

@charlietomo I added a 'release' that can simply be downloaded... that should address point number 3

Rod-Persky commented 5 years ago
Rod-Persky commented 5 years ago

@bradsteels do you have a website that you usually get this info from?

bradsteels commented 5 years ago

i used to get it from Google Finance using Google Docs spreadsheet but recently they stopped updating the mutual fund price histories, so I don't know where to get it now, unfortunately.

Rod-Persky commented 5 years ago

@bradsteels does any provider on the internet show an interactive chart of these mutual funds? If so it should be fairly easy to load it 👍 ... unless:

1) It requires a login 2) It runs on an asp.net server 3) It has some very special state based stuff (ewwwww)

bradsteels commented 5 years ago

it seems that google and yahoo finance do show daily fund prices but do not show historical or chart data now for funds such as CHO102 that I am interested in. i can find historical charts at www.fundlibrary.com without a special login but it may be an asp.net unfortunately.

Rod-Persky commented 5 years ago

Fundlibrary uses non-interactive charts (static images) and asp.net which means getting the data involves pain and image processing haha

charlietomo commented 5 years ago

I understand the app uses either the historical page or the interactive chart. I'm looking at some funds (e.g. GB0007716078.L) and the latest trade on the chart is correct - but there is no history (just zero). The historical data tab shows, somewhat randomly, just over a week in June(!). Investoscope Updater pulls the out of date June data - not the up to date latest trade.

For me, it would be great if we could pull that last trade price, and update Investoscope - understanding that there is no history. My current solution is to have these as manual entries, which get updated manually after I run the updater. Anything I can do to force using the last value from the chart?

Rod-Persky commented 5 years ago

The data seems really sporadic. Is the value meant to be around 2165 or 20? check the historical quote here

charlietomo commented 5 years ago

The data seems really sporadic. Is the value meant to be around 2165 or 20? check the historical quote here

Yes the history is very sporadic. I haven't found a good source for historic. 2,165 GBp (pence) or 21.65 GBP (pound) is correct. That historical link showing 20 is in GBP (not GBp as says on Yahoo page...).