Move to sqlite + pandas

[epogrebnyak]

Need discuss follwoing plan:

1. Store all form 101 form data for selected dates to sqlite database, same cli commands as before:
   • download
   • unpack
   • make csv
   • import csv
2. Retrieve pandas dataframe with contos (columns) by dates (index)
   • for selected bank (regn) regn_101 = get_101_dataframe(regn), where regn is interger
   • for a list of regns regn_101 = get_101_dataframe(regns), where regns is a list of intergers
3. Make balance dataframe for selected bank or group of banks (regn):
   • regn_bal = make_balance(regn)
   • balance_sheet items by column, dates by index
   • use https://github.com/epogrebnyak/cbr-db/blob/pandas/py3/pandas_interface.py as algorithm first approximation
4. Need tests
   • for one regn, eg Sberbank
     • https://github.com/epogrebnyak/cbr-db/tree/master/issues/test_file_selection_(was_stream_src) + may also use http://cbr.ru/credit/101.asp?regnum=1481&when=0&dt=20160201
   • for top-30 banks group of regn http://cbr.ru/analytics/bank_system/svst01122015.pdf:
     • group composition at page 1
     • values at page 2-3
     • aggregation algorith, starting page 4

[epogrebnyak]

top-30 group composition:

top_30_regns = [2590, 1326, 436, 2748, 2289, 30, 2562, 1439, 1460, 1000,
1623, 354, 323, 912, 1978, 3016, 3251, 3292, 2272, 3349, 328, 1481, 1470,
2557, 963, 429, 2209, 1971, 316, 1]

(may be incomplete due to banks M&A)

[alexanderlukanin13]

@epogrebnyak

1. OK
2. Please provide output dataframe example (what columns it will contain?)
3. Please provide output dataframe example (what columns it will contain?)
4. Do you want live tests for this (connecting to real sites, getting real data)?