Back out smf-elements-2.txt updates?

[Bruce-95]

Mel,

I noticed that you backed out the updates to https://climbermel.github.io/SMF_Add-in/Elements/smf-elements-2.txt. And it's not included in the today's Add-in.

Was there a problem with the file?

~ Bruce S

[ClimberMel]

I messed up with putting the file straight into main, so I backed it out then added it to the update branch.  It should be your file in prod now.I'm getting a headache trying to scrape stuff from Yahoo.  I can't seem to find tags to reference... I can see the data, but can't seem to get to it.MelSent from my Galaxy -------- Original message --------From: Bruce S @.> Date: 2024-04-29 4:45 p.m. (GMT-08:00) To: ClimberMel/SMF_Add-in @.> Cc: Subscribed @.***> Subject: [ClimberMel/SMF_Add-in] Back out smf-elements-2.txt updates? (Issue #71) Mel, I noticed that you backed out the updates to https://climbermel.github.io/SMF_Add-in/Elements/smf-elements-2.txt. Was there a problem with the file? ~ Bruce S

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

[Bruce-95]

"I messed up with putting the file straight into main, so I backed it out then added it to the update branch. It should be your file in prod now."

I just downloaded yesterday's addin. It still has the old smf-elements-2.txt file. And the https://climbermel.github.io/SMF_Add-in/Elements/smf-elements-2.txt also shows the old elements. Got more comments re stuff "not working" - post post .

~ Bruce S

[Bruce-95]

I'm working on new fixes to some elements not working in smf-elements-2.txt . Don't update the web site just yet.

~ Bruce S

[ClimberMel]

Crap, I'll check that out.  I was trying to do things to "proper" way and use a separate branch for all the changes and then merge them...ThanksMelSent from my Galaxy -------- Original message --------From: Bruce S @.> Date: 2024-05-01 5:11 p.m. (GMT-08:00) To: ClimberMel/SMF_Add-in @.> Cc: ClimberMel @.>, Comment @.> Subject: Re: [ClimberMel/SMF_Add-in] Back out smf-elements-2.txt updates? (Issue #71) "I messed up with putting the file straight into main, so I backed it out then added it to the update branch. It should be your file in prod now." I just downloaded yesterday's addin. It still has the old smf-elements-2.txt file. And the https://climbermel.github.io/SMF_Add-in/Elements/smf-elements-2.txt also shows the old elements. Got more comments re stuff "not working" - post post . ~ Bruce S

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[ClimberMel]

OK, I'll watch Issue #71 for updates.Sent from my Galaxy -------- Original message --------From: Bruce S @.> Date: 2024-05-01 5:35 p.m. (GMT-08:00) To: ClimberMel/SMF_Add-in @.> Cc: ClimberMel @.>, Comment @.> Subject: Re: [ClimberMel/SMF_Add-in] Back out smf-elements-2.txt updates? (Issue #71) I'm working on new fixes to some elements not working in smf-elements-2.txt . Don't update the web site just yet. ~ Bruce S

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[Bruce-95]

Hi Mel,

burning the midnight oil ....

To see how bad the problem was with "smf-elements-2.txt" I dumped every element in the file into a spreadsheet. Some elements are for a stock, others for a fund. I used the tickers IBM & VFINX to generate data. This is what I found:

Elements - 390

Data returned as "Error" - 41 "Obsolete" - 137 "Not Found" - 25 (varies depending if different tickers used)

This was after I already fixed about 20 elements that were getting "Error". The ones still getting errors are 974 and 1552-1591.

I'll keep going tomorrow but it's going to be slow. A big part of what's left will be harder because of the changes Yahoo made to balance sheet data (nbrs 1552-1591). If would be a big help if we could find the JSON module that has that stuff instead of trying to scrape it.

I could give you what I have so far so you can get it out to people with the stipulation that we're still working on balance sheet info. At least they'll have most of it working.

Let me know what you want to do.

~ Bruce S

[Bruce-95]

PS ... in my previous note when I said elements were fixed, numbers were coming back instead of "Error". I still need to verify the numbers are correct. I'll do that today.

[ClimberMel]

OK, let me know tomorrow and I'll post what you have with a note that it is a work I progress. Thanks, Mel

[Bruce-95]

(Sorry, was away for a couple of days)

Attached is a "some what" updated smf-elements-2.txt file. However ...

I haven't been able to differentiate between the quarter-ending and year-ending balance sheet nbrs . The elements (1552-1591) are for quarter-ending data but the Yahoo balance sheet page defaults to year-ending data. Need to figure out how to get the quarterly page. I'll keep working on it.

Also noticed some peculiar data in the fund performance elements. I'll look at that too.

If you can find the JSON modules for the balance sheet data that would be a help.

~ Bruce S

smf-elements-2.txt

[ClimberMel]

Thanks.  Part of the trouble I was having is that sections are collapsed, or you select a tab, but that no longer changes the URL so there seems no way to go directly to it.Sent from my Galaxy -------- Original message --------From: Bruce S @.> Date: 2024-05-05 3:29 p.m. (GMT-08:00) To: ClimberMel/SMF_Add-in @.> Cc: ClimberMel @.>, Comment @.> Subject: Re: [ClimberMel/SMF_Add-in] Back out smf-elements-2.txt updates? (Issue #71) (Sorry, was away for a couple of days) Attached is a "some what" updated smf-elements-2.txt file. However ... I haven't been able to differentiate between the quarter-ending and year-ending balance sheet nbrs . The elements (1552-1591) are for quarter-ending data but the Yahoo balance sheet page defaults to year-ending data. Need to figure out how to get the quarterly page. I'll keep working on it. Also noticed some peculiar data in the fund performance elements. I'll look at that too. If you can find the JSON modules for the balance sheet data that would be help. ~ Bruce S smf-elements-2.txt

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[Bruce-95]

Mel,

attached is an updated smf-elements-2.txt file. Most of the changes involved changing the function's "type" to 4 to get web pages using the "crumb" processing. Except for the following, the elements return good data. You should release this asap.

However here are some problems I found.

Balance Sheet data Elements: 1552-1583 JSON modules: balanceSheettHistory, balanceSheetHistoryQuarterly

Elements 1552-1583 were pointing to data for the last 4 quarters. But now I have no idea how to grab the "quarterly" web page using Yahoo's new web pages. The only data I found in the JSON modules was the annual and quarterly endDates. There was no other data. According to this site there's suppose to be more.

Income Statement data Elements: 1584-1591, 933-937 JSON modules: incomeStatementHistory, incomeStatementHistoryQuarterly

The elements 1584-1591 were pointing to data for the last 4 quarters. Yahoo's Income Statement page is like the Balance Sheet page. You have to click on a tab to see the quarterly data. Have no idea how to scrape the "quarterly" web page.

The only quarterly data available in the JSON modules for the last 4 quarters were "endDate", "totalRevenue", and "netIncome".

Elements 1584-1587 - were changed to use smfGetYahooJSONField to get the quarterly "endDate"s.

1584;YahooQIS;Quarterly Income Statement -- Date -- FQ1;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.0.endDate.fmt") 1585;YahooQIS;Quarterly Income Statement -- Date -- FQ2;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.1.endDate.fmt") 1586;YahooQIS;Quarterly Income Statement -- Date -- FQ3;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.2.endDate.fmt") 1587;YahooQIS;Quarterly Income Statement -- Date -- FQ4;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.3.endDate.fmt")

Elements 1588-1591 - were changed to "Obsolete" because they weren't in the JSON module nor available to be scraped.

Elements 1592-1598 - are new. These were added for quarterly "totalRevenue" and "netIncome".

1592;YahooQIS;Quarterly Income Statement -- Total Revenue -- FQ1;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.0.totalRevenue.raw") 1593;YahooQIS;Quarterly Income Statement -- Total Revenue -- FQ2;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.1.totalRevenue.raw") 1594;YahooQIS;Quarterly Income Statement -- Total Revenue -- FQ3;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.2.totalRevenue.raw") 1595;YahooQIS;Quarterly Income Statement -- Total Revenue -- FQ4;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.3.totalRevenue.raw") 1596;YahooQIS;Quarterly Income Statement -- Net Income -- FQ1;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.0.netIncome.raw") 1597;YahooQIS;Quarterly Income Statement -- Net Income -- FQ2;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.1.netIncome.raw") 1598;YahooQIS;Quarterly Income Statement -- Net Income -- FQ3;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.2.netIncome.raw") 1599;YahooQIS;Quarterly Income Statement -- Net Income -- FQ4;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.3.netIncome.raw")

Elements 933-937 (incomeStatementHistory) - these weren't changed and are still getting errors. Also these elements don't specify annual or quarterly periods so I don't know what data I would use even if I could.

You could replace -

933;YahooIS;Total Expenses;=smfGetTagContent("https://finance.yahoo.com/quote/~~~~~/financials","span",1,">Total Expenses",,,,1,,4) 934;YahooIS;Operating Expense;=smfGetTagContent("https://finance.yahoo.com/quote/~~~~~/financials","span",1,">Operating Expense",,,,1,,4) 935;YahooIS;Net Income;=smfGetTagContent("https://finance.yahoo.com/quote/~~~~~/financials","span",1,">Net Income",,,,1,,4) 936;YahooIS;Cost of Revenue;=smfGetTagContent("https://finance.yahoo.com/quote/~~~~~/financials","span",1,">Cost of Revenue",,,,1,,4) 937;YahooIS;Total Revenue;=smfGetTagContent("https://finance.yahoo.com/quote/~~~~~/financials","span",1,">Total Revenue",,,,1,,4)

with these giving people better info instead of just "error".

933;YahooIS;Total Expenses;="N/A" 934;YahooIS;Operating Expense;="N/A" 935;YahooIS;Net Income;="See elements 1596-1599." 936;YahooIS;Cost of Revenue;="N/A" 937;YahooIS;Total Revenue;="See elements 1592-1595."

I'll leave it to you to decide if you want to replace these 5 elements.

Observations

I spent hours trying to find what Yahoo did. It seems to me they're making incremental changes to put more info behind a pay wall . It looks like they are using different software to rebuild the web pages with different/missing data (and HTLM code) which is trashing the SMF Addin. If so we'll be forever playing "whack-a-mole", the code will get more bloated, etc. Here's some things I'm throwing out randomly -

• consider rewriting the Addin to use a different, more stable source, i.e. Zack's, Morning Star, Python (yFinance?) etc.
• present the problems to the group and ask for feedback.
• get more members to pitch in and maybe divvy up parts of the code when problems occur.
• stay with Yahoo, but I think will be back here in 3-6 mos. and may also lose availability of more data.
• ???

At this point I need to take a break for a few days. I know this is very time consuming for you too.

Let me know if you have any questions.

~ Bruce S

[ClimberMel]

Thanks for all the work Bruce.  I'll get this posted tomorrow. MelSent from my Galaxy -------- Original message --------From: Bruce S @.> Date: 2024-05-10 5:30 p.m. (GMT-08:00) To: ClimberMel/SMF_Add-in @.> Cc: ClimberMel @.>, Comment @.> Subject: Re: [ClimberMel/SMF_Add-in] Back out smf-elements-2.txt updates? (Issue #71) Mel, attached is an updated smf-elements-2.txt file. Most of the changes involved changing the function's "type" to 4 to get web pages using the "crumb" processing. Except for the following, the elements return good data. You should release this asap. However here are some problems I found. Balance Sheet data Elements: 1552-1583 JSON modules: balanceSheettHistory, balanceSheetHistoryQuarterly Elements 1552-1583 were pointing to data for the last 4 quarters. But now I have no idea how to grab the "quarterly" web page using Yahoo's new web pages. The only data I found in the JSON modules was the annual and quarterly endDates. There was no other data. According to this site there's suppose to be more. Income Statement data Elements: 1584-1591, 933-937 JSON modules: incomeStatementHistory, incomeStatementHistoryQuarterly The elements 1584-1591 were pointing to data for the last 4 quarters. Yahoo's Income Statement page is like the Balance Sheet page. You have to click on a tab to see the quarterly data. Have no idea how to scrape the "quarterly" web page. The only quarterly data available in the JSON modules for the last 4 quarters were "endDate", "totalRevenue", and "netIncome". Elements 1584-1587 - were changed to use smfGetYahooJSONField to get the quarterly "endDate"s. 1584;YahooQIS;Quarterly Income Statement -- Date -- FQ1;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.0.endDate.fmt") 1585;YahooQIS;Quarterly Income Statement -- Date -- FQ2;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.1.endDate.fmt") 1586;YahooQIS;Quarterly Income Statement -- Date -- FQ3;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.2.endDate.fmt") 1587;YahooQIS;Quarterly Income Statement -- Date -- FQ4;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.3.endDate.fmt") Elements 1588-1591 - were changed to "Obsolete" because they weren't in the JSON module nor available to be scraped. Elements 1592-1599 - are new. These were added for quarterly "totalRevenue" and "netIncome". 1592;YahooQIS;Quarterly Income Statement -- Total Revenue -- FQ1;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.0.totalRevenue.raw") 1593;YahooQIS;Quarterly Income Statement -- Total Revenue -- FQ2;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.1.totalRevenue.raw") 1594;YahooQIS;Quarterly Income Statement -- Total Revenue -- FQ3;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.2.totalRevenue.raw") 1595;YahooQIS;Quarterly Income Statement -- Total Revenue -- FQ4;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.3.totalRevenue.raw") 1596;YahooQIS;Quarterly Income Statement -- Net Income -- FQ1;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.0.netIncome.raw") 1597;YahooQIS;Quarterly Income Statement -- Net Income -- FQ2;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.1.netIncome.raw") 1598;YahooQIS;Quarterly Income Statement -- Net Income -- FQ3;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.2.netIncome.raw") 1599;YahooQIS;Quarterly Income Statement -- Net Income -- FQ4;=smfGetYahooJSONField("~","incomeStatementHistoryQuarterly","quoteSummary.result.0.incomeStatementHistoryQuarterly.incomeStatementHistory.3.netIncome.raw") Elements 933-937 (incomeStatementHistory) - these weren't changed and are still getting errors. Also these elements don't specify annual or quarterly periods so I don't know what data I would use even if I could. You could replace - 933;YahooIS;Total Expenses;=smfGetTagContent("https://finance.yahoo.com/quote/~~~~~/financials","span",1,">Total Expenses",,,,1,,4) 934;YahooIS;Operating Expense;=smfGetTagContent("https://finance.yahoo.com/quote/~~~~~/financials","span",1,">Operating Expense",,,,1,,4) 935;YahooIS;Net Income;=smfGetTagContent("https://finance.yahoo.com/quote/~~~~~/financials","span",1,">Net Income",,,,1,,4) 936;YahooIS;Cost of Revenue;=smfGetTagContent("https://finance.yahoo.com/quote/~~~~~/financials","span",1,">Cost of Revenue",,,,1,,4) 937;YahooIS;Total Revenue;=smfGetTagContent("https://finance.yahoo.com/quote/~~~~~/financials","span",1,">Total Revenue",,,,1,,4) with these giving people better info instead of just "error". 933;YahooIS;Total Expenses;="N/A" 934;YahooIS;Operating Expense;="N/A" 935;YahooIS;Net Income;="See elements 1596-1599." 936;YahooIS;Cost of Revenue;="N/A" 937;YahooIS;Total Revenue;="See elements 1592-1595." I'll leave it to you to decide if you want to replace these 5 elements. Observations I spent hours trying to find what Yahoo did. It seems to me they're making incremental changes to put more info behind a pay wall . It looks like they are using different software to rebuild the web pages with different/missing data (and HTLM code) which is trashing the SMF Addin. If so we'll be forever playing "whack-a-mole", the code will get more bloated, etc. Here's some things I'm throwing out randomly -

consider rewriting the Addin to use a different, more stable source, i.e. Zack's, Morning Star, Python (yFinance?) etc. present the problems to the group and ask for feedback. get more members to pitch in and maybe divvy up parts of the code when problems occur. stay with Yahoo, but I think will be back here in 3-6 mos. and may also lose availability of more data. ???

At this point I need to take a break for a few days. I know this is very time consuming for you too. Let me know if you have any questions. ~ Bruce S

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>