Data from Yahoo is correct but one day behind what is available publicly online?

[charlietomo]

Hi @Rod-Persky is the following by design? I have looked at the code but am not a coder so don't fully understand.

I thought it grabbed the data from Yahoo Finance - but I am confused as to the data being a day delayed compared to the web interface or the manually downloaded files from Yahoo.

See below image. Main browser window shows latest close price for VOD.L of 175.02 on Aug 15 and this is the same in the bottom Excel window, showing the CSV which was manually downloaded. However, Investoscope Easy Update (latest version) has just run and updated, with no error for Vodafone. It shows a close price of 178.78 for Aug 14. This is correct, but it is not showing the latest price for 15 Aug which seems to be available?

Is this intended behaviour / all the script can get? Or is there maybe a line offset glitch in the script? Once again, the data in Investoscope is correct, just not as up to date as it appears it could be. Thanks in advance for continuing the great work on this and ensuring Investoscope limps on :-)

[Rod-Persky]

@charlietomo welcome to the fun world of coding haha! So the important part of the yahoo finance query is Here which asks yahoo for data from today back 3 years. (You may recognise that 3 year bit from the other discussion where you saw that the updater only grabs relatively recent data).

Actually, the code dumps everything it parses into a folder called csv_data. Perhaps you can find a file that should be called vod_l.csv and also take a screenshot again with the data after running the update process?

I will have to check if I technically use the Historic data, because there is a second method called yahoo_json.py which actually grabs data that is presented in the charts. The yahoo_json.py method is preferred as it is far easier to parse for python, and a lot easier to produce (for yahoo).

[charlietomo]

Haha yes it seems a fun world!

I have a file called vod_l.csv which was updated back in March 2018 (?!), and I have a file called VOD_L_yv2.csv which was updated recently i.e. when I ran the updater. That file shows the same as Investoscope (delayed), which is different to online (historic or charts).

The following screenshot shows Yahoo - both the download section and the charts (is this where the yahoo_json.py gets info from?) Both of those show the latest (as writing close of market on at 176.60 on 20 Aug). The CSV in excel shows the same as Investoscope after an update - 175.78 as close of market on 17 August.

I'm wondering if this is a time zone issue...related to your comments in another issue re checking for time etc. I'm in a different timezone (i.e. not London, where VOD.L is based). Potentially related or just a red herring!?

[KhaineBOT]

I have the same issue. I ran Investoscope-updater today (28th) at 7PM. The historical data only goes up to 24 August. If I check the csv files (i.e anz_asx.csv) the data only goes up to 24 August.

[charlietomo]

@KhaineBOT what time zone are you in and what market are you updating? Are you in Australia and updating Australian instruments (e.g. ANZ)? Wondering if it is a timezone mismatch (but my coding skills not up to troubleshooting very well!).

[KhaineBOT]

I am in Australia (EST) and updating Australian instruments. Its ~9am on the 29th EST, and I just updated Investoscope. It now has historical data up to the 27th (so missing Tuesday).

I had a quick look at your code, and couldn't see anything that looked obviously wrong. I need to see what data yahoo returns to determine the underlying issue, I didn't have time to do any of that last night.

[Rod-Persky]

There isn't any problem with time zone, Yahoo uses seconds since Unix Epoch which is used within the code to get the as implimented here. However, for AU stocks a QuoteAPI is used, which may be one possible cause.

[charlietomo]

Hi @Rod-Persky I'm seeing the issue with UK stocks (on LSE exchange). Does that use a different technique or is there a way I can test things further to try and resolve it?

It makes it much harder for me to cross reference against broker and to get a current picture.

Can you replicate the issue on your setup?

[charlietomo]

I just checked and I'm mainly updating LSE instruments (UK). I get trading price from Friday 24 August - yet it is 6am on Wednesday 29 August, and Yahoo data is showing price from (close of business) on Tuesday 28 August. How can we assist fixing this - screenshots or beer/coffee $?!

[Rod-Persky]

The recieved ticks are unix timestamps, and they are formatted into the local time zone here. There may be a mismatch between how Investoscope anticipates the time zones and the code gives investoscope the time data.

{  
  "chart":{  
    "result":[  
      {  
        "meta":{  
          "currency":"GBp",
          "symbol":"LLOY.L",
          "exchangeName":"LSE",
          "instrumentType":"EQUITY",
          "firstTradeDate":820134900,
          "gmtoffset":3600,
          "timezone":"BST",
          "exchangeTimezoneName":"Europe/London",
          "chartPreviousClose":64.91,
          "priceHint":2,
          "currentTradingPeriod":{  
            "pre":{  
              "timezone":"BST",
              "start":1535523300,
              "end":1535526000,
              "gmtoffset":3600
            },
            "regular":{  
              "timezone":"BST",
              "start":1535526000,
              "end":1535556600,
              "gmtoffset":3600
            },
            "post":{  
              "timezone":"BST",
              "start":1535556600,
              "end":1535559300,
              "gmtoffset":3600
            }
          },
          "dataGranularity":"1d",
          "validRanges":[  
            "1d",
            "5d",
            "1mo",
            "3mo",
            "6mo",
            "1y",
            "2y",
            "5y",
            "10y",
            "ytd",
            "max"
          ]
        },
        "timestamp":[  
          1485763200,
          1485849600,
          1485936000,
          1486022400,
          1486108800,
          1486368000,
          *SNIP*
        ],
        "events":{  
          "dividends":{  
            "1502348400":{  
              "amount":1.0,
              "date":1502348400
            },
            "1491462000":{  
              "amount":1.7,
              "date":1491462000
            }
          }
        },
        "indicators":{  
          "quote":[  
            {  
              "open":[  
                65.4800033569336,
                65.0,
                65.6500015258789,
                65.19999694824219,
                65.0,
                66.31999969482422,
                *SNIP*
              ],
              "low":[  
                64.83000183105469,
                64.86000061035156,
                65.4000015258789,
                64.55000305175781,
                64.7239990234375,
                66.0199966430664,
                *SNIP*
              ],
              "high":[  
                65.73999786376953,
                65.94000244140625,
                66.19000244140625,
                65.55999755859375,
                66.13999938964844,
                66.70999908447266,
                *SNIP*
              ],
              "close":[  
                64.91000366210938,
                64.93000030517578,
                65.41999816894531,
                64.88999938964844,
                65.8499984741211,
                66.05999755859375,
                *SNIP*
              ],
              "volume":[  
                153568514,
                152232343,
                160896092,
                137384817,
                160854122,
                117547095,
                *SNIP*
              ]
            }
          ],
          "adjclose":[  
            {  
              "adjclose":[  
                59.32339859008789,
                59.3416748046875,
                59.78950119018555,
                59.305118560791016,
                60.18248748779297,
                60.374412536621094,
                 *SNIP*
              ]
            }
          ]
        }
      }
    ],
    "error":null
  }
}

[charlietomo]

There may be a mismatch between how Investoscope anticipates the time zones and the code gives investoscope the time data.

@Rod-Persky I think you are correct with this. Some other information based on recent testing. I'm now running on UK timezone on my Mac (and am also in the UK, in case there is any geo-locating via IP or similar going on).

Its 11am on 6 September, and I run Investoscope updater and I get all my LSE instruments updated to close on 5 September - i.e. what I would hope for. My previous post when this didn't work would have been in a different time zone (and physical country).

However now I see that my ASX listed instruments are delayed, along the lines of my first post i.e. its 8pm on 6 September, Yahoo has close of business for 6 September, but Investoscope only shows data until 4 September.

I'm not sure how or why the local timezones on my Mac seem to be impacted what is downloaded - any ideas?

[KhaineBOT]

I think I've found the error:

for idx in range(0,len(data['close'])-1):

I think it should be:

for idx in range(0,len(data['close'])):

otherwise you are asking for the data for all the days to the last close less one day. Someone who knows the code and the Quotes API better should confirm this. @Rod-Persky you mentioned that the ASX uses Quote API, and so shouldn't be impacted by the yahoo code.

[KhaineBOT]

Turns out changing

for idx in range(0,len(data['close'])-1):

to

for idx in range(0,len(data['close'])):

didn't solve the bug. Can anyone confirm that it isn't QuoteAPI providing the data delayed by one day?

[Rod-Persky]

@KhaineBOT yes, that would result in an off by one error 👍 @charlietomo I'm 80% sure that the timezone formatting on the mentioned line is a primary cause of the delay. I still need to confirm how investoscope handles time zones with the AppleScript.

[KhaineBOT]

@Rod-Persky even with that change, I still see the historical quotes being delayed by a day. For example, if I run the script after the close it only updates to the close of the day before.

[charlietomo]

OK @Rod-Persky let us know if I can be any help testing.

[charlietomo]

Some progress for me.

Its 23:00 on Sunday 23rd. I run Investoscope updater and I get results for LSE stocks until close of play on Thursday 20th. I should (and Yahoo has) got them up to Friday 21st. I've deleted my state.p and tried again with the same result. I'm in the UK and on UK timezone laptop setting so shouldn't be confused.

I take @KhaineBOT tips from earlier and edit my yahoo_json.py and edit line 81 from

for idx in range(0,len(data['close'])-1):

to

for idx in range(0,len(data['close'])):

I delete my state.p and re-run Investoscope updater and ALL MY LSE STOCKS ARE CORRECTLY UPDATED TO CLOSE OF PLAY FRIDAY 21ST! This is a great result for me. Without testing further I hope it fixes this for other situations (e.g. machines with different timezones), and perhaps (hopefully) the timezones differing was a red herring @Rod-Persky ?

[charlietomo]

@KhaineBOT nice fix. I tested things with using different timezones and the fix seems to work so I think the timezone issue was a red herring and not relevant.