Feng-CityUHK / EquityCharacteristics

Calculate U.S. equity (portfolio) characteristics
https://feng-cityuhk.github.io/EquityCharacteristics/
82 stars 67 forks source link

Characteristic .py files too many connections #27

Open ThomasMConnors opened 3 months ago

ThomasMConnors commented 3 months ago

A connection to WRDS is created in the main Python process -- and also -- in each pool process. With a large pool e.g. 20, errors of a PostgreSQL too many database connections can occur. Other redundant code is also processed unnecessarily.

There is a test for name == 'main' and this also needs to be expanded to top and bottoms of code that only needs execution in the main process.

A judicious use of a conn.close() is also warranted to free the connection(s) for other characteristic runs in parallel.

mwri469 commented 1 week ago

I am having this issue due to restricted compute memory, is there ant way you can be more specific for me? By test for main do you just mean wrapping conn = wrds.Connection() to before def_[characteristic]() in a if name == 'main'?

Also for conn.close(), where would be the most appropriate location to add this if changing locally?

TIA