search

caiobran / mstables

MorningStar.com scraper that consolidates tens of thousands of financial records into a SQLite relational database. Class 'dataframes' easily converts the SQLite data into pandas DataFrames (see Jupyter notebook for examples)
MIT License
174 stars 45 forks source link

Access to ciaobran #8

Open chrisidso opened 2 years ago

chrisidso commented 2 years ago

Hi Andrew, Just tried to clone ciaobran and saw an error message that said my login did not allow me access to ciaobran.

I did check to see that I had all the python modules I needed, and I did have them.

how to I get access to ciaobran?

Chris

datatalking commented 2 years ago

Chris so what you will need to do is copy this to your GitHub and pull it down from there.

Let me know if that makes sense

chrisidso commented 2 years ago

That makes sense but I have no idea how to do that. Also, github has changed their access (since the last time I accessed my projects) so it now requires ssh keys. I was experimenting with that yesterday and got nowhere. So at this point the best I can do is clone the repository down to my machine and then run it, and play with it.
How do I copy it to my own github account?

Haven't tried on windows yet. I will need to install all of the python stuff mentioned in the readme including jupyter notebook in order to do that.

chrisidso commented 2 years ago

Andrew, OK. Figured out how to copy to my github. Then was able to pull it down to my linux machine.

chrisidso commented 2 years ago

Andrew, Working on this now. Sorry for the delay. I had a busy weekend and a small crisis to deal with this week. Working on a test plan now, then testing on linux.

chrisidso commented 2 years ago

OK. Andrew. I have some findings now. I put it through three rounds of tests. And I was only looking at the UI, and not so much at the underlying files/directories. Round 1: Tried options 0,3.4,5, and 6 without first creating the tables. As expected all of them generated errors. No database or no tables to work with. In all of these cases main.py broke, and popped me back to the command line. It should not have done this. Should be able to handle an error and display a suitable error message and keep main.py running. Also the error message should be friendly to the user, instead of being the cryptic python error message. Probably need two levels of menus - one with just options 1 and 2 to start and after that add the other options. Round 2: Option 1 first, then 0,1,3,4,5, and 6. With the tables created all of the other options work, except #6. Executing option 1 after creating the tables works too. It seems happy recreating all of the tables. Option 6 though does not work. And I could not get it to work at all. You'd think that you should type in the name of the backup file, so it can create it, but I got an error regardless of whether I typed the existing name or a new one. Found something else at this point too. So, option 1, then 0 and change the name of the database. Then option #6. This will cause main.py to quit and pop you back to the command line. If you run main.py again, you will see that the database name has reverted to the default name (mstables.mysqlite). It should keep the new name though, until you decide to delete everything and start over.
I noticed that in some of the error messages it actually displays a list of the database names (contains default name and new ones). So it looks like when you change the database name it adds the new name to a list, rather than replacing the old name with the new one.

Round 3: Option 1, then option 2. Then 0,3,4,5,6 All the options work. #4 though displays a 'Goodbye' message and main.py quits, which is different than what happened in round 1. As with round 1, #6 did not work at all.

Then I tried deleting everything and then Option #2. As expected this generates an error message because there are no tables.

I just found something else. Option 1, then 2, and 2 again. I asked for 10 records in both cases. Seems to work, but also seems to have put the scraper into a weird state such that: I executed options 3,4,5 to delete everything. Now , when I run main.py and choose option #4, it works. It deletes the tables, without an error message, even though I already deleted the tables. I can execute option 4 as many times as I want to and it will keep deleting the tables. Option #3 fails though because there are no tables. ?????

datatalking commented 2 years ago

@chrisidso these are good notes thank you for the process thinking, its helpful. When you put in the notes '#4' or '#6' are you actually referring to those issue numbers? Currently when you reference #3 it references (issue #3 on my screen a link pops up Fail to initialize database, Win10 #3

Instead i'm thinking you are referring to the general repo general menu here [?] img

So there was an update by a user pr 05957ad are you speaking to the same issue or different?

So with round 1 there should be a database that was created 05957ad was a PR that should have fixed this (should have I think)

Rund 2

With 'Option 1, then 2, and 2 again' the scraper itself doesn't access morningstar and I am looking at the option of adding in the data feed from yahoo, tiingo and others via pandadata reader but that is a sort of seperate PR with @caiobran permission or approval.

Round 3

So when you do '#6' menu option (is there a better way we should note that?) we should get feedback from @caiobran on how the backup is supposed to work. I've not dug deeper into how that was setup or maybe it needs to be address in another issue?

chrisidso commented 2 years ago

Andrew, Correct. I was referring to the numbers of the menu items and not the existing issues.

I don't know if update 05957ad addresses the same issue. It may well be though.

About failure of option #6. I would create a separate issue for that.

chrisidso commented 2 years ago

Andrew, So I am still trying to get set up on windows. I have installed everything I need (I think) but I haven' t figured out yet how to get Jupyter Notebook to run. I'll keep you posted. And when I get it working I will test it. Chris

chrisidso commented 2 years ago

Andrew,

You are right. Issue #4 does address the round one issue I mentioned above.
Do you want me to create a separate issue for the Menu Option #6 failure?

datatalking commented 2 years ago

Hi Chris, I'm just seeing this now.

Can you pull or access the data from my githubrepo for https://github.com/caiobran/mstables/issues/8#issuecomment-1028237736

On Wed, Feb 2, 2022 at 10:35 AM chrisidso @.***> wrote:

That makes sense but I have no idea how to do that. Also, github has changed their access (since the last time I accessed my projects) so it now requires ssh keys. I was experimenting with that yesterday and got nowhere. So at this point the best I can do is clone the repository down to my machine and then run it, and play with it. How do I copy it to my own github account?

Haven't tried on windows yet. I will need to install all of the python stuff mentioned in the readme including jupyter notebook in order to do that.

— Reply to this email directly, view it on GitHub https://github.com/caiobran/mstables/issues/8#issuecomment-1028237736, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC2ECWIEUA2W67NIILDMSE3UZF2N3ANCNFSM5M4YDYDA . You are receiving this because you commented.Message ID: @.***>

datatalking commented 2 years ago

Chris, please do post a seperate issue, I'm waiting to hear back from @caiobran

datatalking commented 2 years ago

@chrisidso I found the link I was looking for, https://github.com/joe-wojniak/PythonForFinance is the data source that can be used locally since the scrape isnt working any more.

I'll likely hear back from @caiobran soon, but what I am thinking of is creating a 'parsing tool' unless its already built for either IEX, Stooq or PandasData reader.

https://www.mssqltips.com/sqlservertip/6963/download-stooq-historical-stock-price-volume-data-sql-server/

I am still researching these

chrisidso commented 2 years ago

Hi Andrew, I just created an issue for the menu #6 error. I included the entire error message too.

chrisidso commented 2 years ago

Still having difficulty on windows too. Cannot get jupyter to run and was unable to run main.py too. My computer did not seem to know what py or python meant. Still working on it.

chrisidso commented 2 years ago

I have not tried to pull the data from your github repo yet. Did not see the link to it in your earlier post. Can you re-post the link?

datatalking commented 2 years ago

As far as running .py jupyter notebooks usually want to run a .ipynb, what IDE are you using? Was it the mstables.db for sqlite, the gold stuff or ? Remind me what data i was supposed to send you? I've been connecting 'investpy', 'pandasdataframe' and 'mstables' so my brain is a bit full these days.

chrisidso commented 2 years ago

I have had a very busy week (whew... job hunting, interviewing etc). I wanted to send a message so that you would know that I am still here and still interested in helping out with this project. I probably will not have time to look at it tomorrow, and maybe not on Sunday either. But I will carve out some time here soon. And I will likely have some questions too. :)

datatalking commented 2 years ago

Hey Chris that is fine, I'm building a Django front end site for a startup project. I'll be checking this daily for a while and doing a lot of coding over the break. Let me know if you would like a referral for your job. Could say you have been "Helping with database testing, front end work and backend integration with SQL and Python"

datatalking commented 1 year ago

@chrisdso Have you been able to get this working ?

chrisidso commented 1 year ago

Hey Andrew! Sorry... No. I have not been able to get it working. In fact I have not looked at it for a long time. An update - I have had difficulty finding employment - I was a contractor over the summer (SDET) but that job ended earlier than I thought and I have been trying to get another job since then. I have a job now - but as cashier at a local retailer (seasonal - and evenings and weekends) - and it is exhausting. They have me working almost 40 hours a week so I do not have time to do much of anything else right now.

datatalking commented 1 year ago

@chrisidso It is good to hear from you, my apologies for the long delay as I somehow never got a notification from GitHub of your reply.

While we did lose the MS data-feed and to the MSratio value-feed, but through a contact of mine, we were to scavenge/replicate what looks to be all of the MSratio data via found formulas.

We were able to re-source (about) 68% of the data so far in whole and most of the rest should be able to be patched with other api's patched into a secondary database and now I'm testing, building upon your work in parse.py

I could use your big brain to expand upon the mstables database, menue and to include a few API's, a web-scraper and curiosity about Forex, ETF or Crypto options. We could use GitHub project boards for feature planning and other features, despite the drop in business I enjoy building on your work and gotta keep practicing these dev ops skills so they stay sharp.

chrisidso commented 1 year ago

Hi Andrew, Sorry for the long delay in getting back to you. I saw your email in one of my email accounts right after you sent it and just was not sure how I should respond. I am sure my software development skills are rusty by now. Not sure why I see emails immediately and you do not. ???
Anyway, I am interested in helping out with this project. I am starting another job very soon now and have lots of training/prep to go through so I do will not have much time to work on this project. Can you enlighten me as to what you want to do with the menu and the database?
Sorry to hear that you lost and then recovered some of your data. ): That must have been quite a challenge to fix!
Chris