search

carstenf / Security-Master

database for securities, with import from Quandl / Yahoo and ingest to zipline
11 stars 3 forks source link

result.dimension error in quandl_to_db #1

Closed ghost closed 4 years ago

ghost commented 4 years ago

at line 495 in the function fill_fundamental_data, i get an error: 'DataFrame' object has no attribute 'dimension'.

` result3 = result.loc[result.dimension == 'ARY'] result2 = result.loc[result.dimension == 'ARQ' ]

result1 = result.loc[result.dimension == 'ART' ]`

what do you mean by dimension? shape/size/dimn

carstenf commented 4 years ago

are you using Quandl withs this package Core US Fundamentals Data PUBLISHED BY SHARADAR?

It looks like you don't have fundamental data. Dimension is ARQ, ARY, ART, MRT.... it's the reporting period AS REPORTED QUATERLY ARQ

carstenf commented 4 years ago

pls check link https://www.quandl.com/databases/SF1/data and see the header of the Core US Fundamentals https://www.quandl.com/tables/SF1/SHARADAR-SF1 with dimension.

carstenf commented 4 years ago

can I close this?

ghost commented 4 years ago

yes, the fundamental data was't available for free so i got the error, thanks.

carstenf commented 4 years ago

ok, good to know, but you can use yahoo as well to fill the database. the only issue is the ticker list. but you can download a ticker list from quandl for free from that bundle I mentioned. than generate the ticker list, I put an example in the repo. than use it with the yahoo_to_db.py

if you want to use the free quandl, you need to comment some call in the main, just at the end of the file quandl_to_db.py

or you get the full package for one month for around 99$....and you can use this database with the alphacompiler to use fundamental data with zipline, just building it :)

ghost commented 4 years ago

Hi thanks for the support. I'm just starting out with algorithmic trading and want to test some things before paying for a vendor. Does the full package and quandl_to_db.py include delisted stocks?

On Wed, Apr 8, 2020 at 7:12 PM carstenf notifications@github.com wrote:

ok, good to know, but you can use yahoo as well to fill the database. the only issue is the ticker list. but you can download a ticker list from quandl for free from that bundle I mentioned. than generate the ticker list, I put an example in the repo. than use it with the yahoo_to_db.py

if you want to use the free quandl, you need to comment some call in the main, just at the end of the file quandl_to_db.py

or you get the full package for one month for around 99$....and you can use this database with the alphacompiler to use fundamental data with zipline, just building it :)

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/carstenf/Security-Master/issues/1#issuecomment-611050501, or unsubscribe https://github.com/notifications/unsubscribe-auth/APDDZDJOX7P6ALHK2OGUEQLRLSPALANCNFSM4MDRAWZQ .

carstenf commented 4 years ago

yes! yahoo, I believe does not, or not for many years. I got the one from Quandl for 20years, I think you can get shorter period for less. which backtester you are planing to use?

carstenf commented 4 years ago

of course there are better data than Quandl, like factset, Bloomberg, Morningstar but the cost a fortune...

ghost commented 4 years ago

99$ isn't really that expensive so I will think ill go for it. can i pay for a month download the data to the database and then not continue with them? or do i have to delete the data?

On Wed, Apr 8, 2020 at 7:27 PM carstenf notifications@github.com wrote:

of course there are better data than Quandl, like factset, Bloomberg, Morningstar but the cost a fortune...

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/carstenf/Security-Master/issues/1#issuecomment-611058541, or unsubscribe https://github.com/notifications/unsubscribe-auth/APDDZDP572RBVFV3TJQTE2TRLSQY3ANCNFSM4MDRAWZQ .

carstenf commented 4 years ago

can i pay for a month download the data to the database and then not continue with them?

thats what I did.

which backtester you are going to use? zipline?

ghost commented 4 years ago

Yes, Zipline looks really great and simple to understand. I'm planning on backtesting some strategies with it and then connect my brokerage(Interactive brokers) and update the database with their data.

On Wed, Apr 8, 2020 at 7:44 PM carstenf notifications@github.com wrote:

can i pay for a month download the data to the database and then not continue with them?

thats what I did.

which backtester you are going to use? zipline?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/carstenf/Security-Master/issues/1#issuecomment-611066712, or unsubscribe https://github.com/notifications/unsubscribe-auth/APDDZDKNY763NN2OFGWKTU3RLSSVLANCNFSM4MDRAWZQ .

carstenf commented 4 years ago

ok, first zipline only works with price, no fundamentals. -> for this I found a solution > alphacompiler second, you can't connect it to a broker -> for this not yet -> did you found something? did you already install zipline?...it was not that easy for me

carstenf commented 4 years ago

...are you using Mac or windows?

ghost commented 4 years ago

Yep I'm kind of working with 2 friends and it was really pain in the ass to get the environment working but it's worth it. I'm working on windows, did it with anaconda and it really simplified things.

On Wed, Apr 8, 2020, 8:28 PM carstenf notifications@github.com wrote:

...are you using Mac or windows?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/carstenf/Security-Master/issues/1#issuecomment-611088745, or unsubscribe https://github.com/notifications/unsubscribe-auth/APDDZDORU5LPKCPSRBAROALRLSX2BANCNFSM4MDRAWZQ .

carstenf commented 4 years ago

...ok, i was thinking to put everything into dockers, but it will probably only work on the same operating system, I'm on Mac

carstenf commented 4 years ago

basically you need:

ghost commented 4 years ago

Hi, hope you are doing well. I decided to buy Sharadar bundle so i try to load the data into my database and i get comments that some price data is missing. in your load did you encounter this too?

On Wed, Apr 8, 2020 at 9:13 PM carstenf notifications@github.com wrote:

basically you need:

  • zipline
  • an interface between data vendor and zipline -> can use my database
  • an additional tool to get fundamental data running on zipline -> alphacompiler, and I need to publish my bridge between alphaxompiler and my database (with this you get fundamental data, sector code and S&P500 member filter
  • than you need a bridge to Interactive brokers-> im still looking as well... ..and if everything is ready, someone needs to put it into dockers...

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/carstenf/Security-Master/issues/1#issuecomment-611111099, or unsubscribe https://github.com/notifications/unsubscribe-auth/APDDZDPQ65INVUOT5FVOAQLRLS5DJANCNFSM4MDRAWZQ .

carstenf commented 4 years ago

Yes, that’s normal. The program first reads the list of tickets from quandl. Based on that list, it reads the price data. If that ticker is not in the list of price data included, you get the massage. Was it difficult until here?

Best Carsten

Am 18.04.2020 um 15:54 schrieb MasterKaido notifications@github.com:

 Hi, hope you are doing well. I decided to buy Sharadar bundle so i try to load the data into my database and i get comments that some price data is missing. in your load did you encounter this too?

On Wed, Apr 8, 2020 at 9:13 PM carstenf notifications@github.com wrote:

basically you need:

  • zipline
  • an interface between data vendor and zipline -> can use my database
  • an additional tool to get fundamental data running on zipline -> alphacompiler, and I need to publish my bridge between alphaxompiler and my database (with this you get fundamental data, sector code and S&P500 member filter
  • than you need a bridge to Interactive brokers-> im still looking as well... ..and if everything is ready, someone needs to put it into dockers...

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/carstenf/Security-Master/issues/1#issuecomment-611111099, or unsubscribe https://github.com/notifications/unsubscribe-auth/APDDZDPQ65INVUOT5FVOAQLRLS5DJANCNFSM4MDRAWZQ .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

ghost commented 4 years ago

thought about adding insider transactions and funds prices that comes with shradar bundle?

On Sat, Apr 18, 2020 at 8:24 PM carstenf notifications@github.com wrote:

Yes, that’s normal. The program first reads the list of tickets from quandl. Based on that list, it reads the price data. If that ticker is not in the list of price data included, you get the massage. Was it difficult until here?

Best Carsten

Am 18.04.2020 um 15:54 schrieb MasterKaido notifications@github.com:

 Hi, hope you are doing well. I decided to buy Sharadar bundle so i try to load the data into my database and i get comments that some price data is missing. in your load did you encounter this too?

On Wed, Apr 8, 2020 at 9:13 PM carstenf notifications@github.com wrote:

basically you need:

  • zipline
  • an interface between data vendor and zipline -> can use my database
  • an additional tool to get fundamental data running on zipline -> alphacompiler, and I need to publish my bridge between alphaxompiler and my database (with this you get fundamental data, sector code and S&P500 member filter
  • than you need a bridge to Interactive brokers-> im still looking as well... ..and if everything is ready, someone needs to put it into dockers...

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub < https://github.com/carstenf/Security-Master/issues/1#issuecomment-611111099 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/APDDZDPQ65INVUOT5FVOAQLRLS5DJANCNFSM4MDRAWZQ

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/carstenf/Security-Master/issues/1#issuecomment-615907524, or unsubscribe https://github.com/notifications/unsubscribe-auth/APDDZDLG5GEBWSUURUZ5USTRNHO3NANCNFSM4MDRAWZQ .

carstenf commented 4 years ago

 thought about adding insider transactions -> is that included? Which table?

and funds prices that comes with shradar bundle? -> you can include them already. Try to understand how the quandl_to_db works. Funds are already in the ticker list, that’s the security table. Than you need to add to the code, that it needs as well to include the ttable for funds, it’s probably SF3???, in the section were you read the pricing. Now it reads only SF1. Later you have to include this as well in the header of the db_to_bundle

I hope it’s not too difficult

Am 19.04.2020 um 10:41 schrieb MasterKaido notifications@github.com:

 thought about adding insider transactions and funds prices that comes with shradar bundle?

On Sat, Apr 18, 2020 at 8:24 PM carstenf notifications@github.com wrote:

Yes, that’s normal. The program first reads the list of tickets from quandl. Based on that list, it reads the price data. If that ticker is not in the list of price data included, you get the massage. Was it difficult until here?

Best Carsten

Am 18.04.2020 um 15:54 schrieb MasterKaido notifications@github.com:

 Hi, hope you are doing well. I decided to buy Sharadar bundle so i try to load the data into my database and i get comments that some price data is missing. in your load did you encounter this too?

On Wed, Apr 8, 2020 at 9:13 PM carstenf notifications@github.com wrote:

basically you need:

  • zipline
  • an interface between data vendor and zipline -> can use my database
  • an additional tool to get fundamental data running on zipline -> alphacompiler, and I need to publish my bridge between alphaxompiler and my database (with this you get fundamental data, sector code and S&P500 member filter
  • than you need a bridge to Interactive brokers-> im still looking as well... ..and if everything is ready, someone needs to put it into dockers...

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub < https://github.com/carstenf/Security-Master/issues/1#issuecomment-611111099 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/APDDZDPQ65INVUOT5FVOAQLRLS5DJANCNFSM4MDRAWZQ

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/carstenf/Security-Master/issues/1#issuecomment-615907524, or unsubscribe https://github.com/notifications/unsubscribe-auth/APDDZDLG5GEBWSUURUZ5USTRNHO3NANCNFSM4MDRAWZQ .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.