search

WLM1ke / poptimizer

Оптимизация долгосрочного портфеля акций
The Unlicense
152 stars 28 forks source link

Отсутствие котировок #67

Closed RomaKoks closed 2 years ago

RomaKoks commented 2 years ago 
Traceback (most recent call last):
  File "pandas\_libs\index.pyx", line 460, in pandas._libs.index.DatetimeEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 2131, in pandas._libs.hashtable.Int64HashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 2140, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 1636416000000000000

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\indexes\base.py", line 3361, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas\_libs\index.pyx", line 429, in pandas._libs.index.DatetimeEngine.get_loc
  File "pandas\_libs\index.pyx", line 462, in pandas._libs.index.DatetimeEngine.get_loc
KeyError: Timestamp('2021-11-09 00:00:00')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\indexes\datetimes.py", line 702, in get_loc
    return Index.get_loc(self, key, method, tolerance)
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\indexes\base.py", line 3363, in get_loc
    raise KeyError(key) from err
KeyError: Timestamp('2021-11-09 00:00:00')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "PATH_TO\PycharmProjects\poptimizer\poptimizer\portfolio\portfolio.py", line 151, in price
    price = price.loc[self.date]
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\indexing.py", line 931, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\indexing.py", line 1164, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\indexing.py", line 1113, in _get_label
    return self.obj.xs(label, axis=axis)
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\generic.py", line 3773, in xs
    loc = index.get_loc(key)
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\indexes\datetimes.py", line 704, in get_loc
    raise KeyError(orig_key) from err
KeyError: Timestamp('2021-11-09 00:00:00')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "PATH_TO\PycharmProjects\poptimizer\optimize.py", line 12, in opt
    optimize(date, ports=ports)
  File "PATH_TO\PycharmProjects\poptimizer\poptimizer\__main__.py", line 24, in optimize
    print(opt.portfolio)
  File "PATH_TO\PycharmProjects\poptimizer\poptimizer\portfolio\portfolio.py", line 66, in __str__
    self._positions_stats(),
  File "PATH_TO\PycharmProjects\poptimizer\poptimizer\portfolio\portfolio.py", line 91, in _positions_stats
    weights = self.weight.iloc[:-2]
  File "PATH_TO\PycharmProjects\poptimizer\poptimizer\portfolio\portfolio.py", line 176, in weight
    value = self.value
  File "PATH_TO\PycharmProjects\poptimizer\poptimizer\portfolio\portfolio.py", line 165, in value
    value = self.price * self.shares
  File "PATH_TO\PycharmProjects\poptimizer\poptimizer\portfolio\portfolio.py", line 153, in price
    raise config.POptimizerError(
poptimizer.config.POptimizerError: Для даты 2021-11-09 отсутствуют исторические котировки

Кроме как удалением всех котировок, можно как-то лечить? Запуск переобучения моделей почему-то не подтягивает свежие котировки. Ошибка возникает при оптимизации. Видел её раньше, приходилось удалять все котировки (включая индексы и misc) и качать заново.

WLM1ke commented 2 years ago

Достаточно в misc удалить документ trading_dates - тогда он обновит все котировки.

RomaKoks commented 2 years ago

Понял, спасибо! А причину её возникновения не знаете? Если нет, то может быть обернуть value = self.price * self.shares в try catch и в нём удалить trading_dates и запустить загрузку котировок?

WLM1ke commented 2 years ago

Причина - по каким-то причинам была прервана загрузка данных. Вариантов основных два - в ручную грохнули программу во ремня загрузки или произошла какая-то ошибка во время загрузки. Оборачивать плохо, так если произошла ошибка при загрузке, то обычно надо что-то подшаманить с загрузчиками данных.

RomaKoks commented 2 years ago

Теперь, кажется, именно при загрузке произошла:

INFO:EventBus:DateCheckRequired(timestamp=datetime.datetime(2021, 11, 12, 5, 16, 47, 965688))
INFO:TradingDatesGateway:Загрузка данных по торговым дням
INFO:Mapper:Сохранение ID(package='data', group='trading_dates', name='trading_dates')
INFO:EventBus:TradingDayEnded(date=datetime.date(2021, 11, 11))
INFO:CPIGateway:Загрузка инфляции
INFO:SmartLabGateway:Загрузка данных
INFO:RFGateway:Загрузка данных
INFO:USDGateway:(2021-11-11, 2021-11-11)
INFO:MOEXStatusGateway:Загрузка данных
INFO:Mapper:Сохранение ID(package='data', group='risk_free', name='risk_free')
INFO:Mapper:Сохранение ID(package='data', group='usd', name='usd')
Traceback (most recent call last):
  File "PATH_TO\poptimizer\loop.py", line 5, in <module>
    from poptimizer.__main__ import evolve
  File "PATH_TO\poptimizer\poptimizer\__main__.py", line 4, in <module>
    from poptimizer.data.views import div_status
  File "PATH_TO\poptimizer\poptimizer\data\views\div_status.py", line 9, in <module>
    from poptimizer.data.app import bootstrap, viewers
  File "PATH_TO\poptimizer\poptimizer\data\app\bootstrap.py", line 47, in <module>
    BUS, VIEWER = start_app()
  File "PATH_TO\poptimizer\poptimizer\data\app\bootstrap.py", line 42, in start_app
    bus.handle_event(event)
  File "PATH_TO\poptimizer\poptimizer\shared\app.py", line 74, in handle_event
    loop.run_until_complete(self._handle_event(event))
  File "PATH_TO\anaconda3\envs\finan\lib\asyncio\base_events.py", line 642, in run_until_complete
    return future.result()
  File "PATH_TO\poptimizer\poptimizer\shared\app.py", line 89, in _handle_event
    pending |= self._create_tasks(task.result())
  File "PATH_TO\poptimizer\poptimizer\shared\app.py", line 100, in _handle_one_command
    return await self._event_handler.handle_event(event, repo)
  File "PATH_TO\poptimizer\poptimizer\data\domain\handlers.py", line 69, in trading_day_ended
    *itertools.chain.from_iterable(await asyncio.gather(*aws)),
  File "PATH_TO\poptimizer\poptimizer\data\domain\handlers.py", line 29, in _load_by_id_and_handle_event
    return await table.handle_event(event)
  File "PATH_TO\poptimizer\poptimizer\data\domain\tables\base.py", line 71, in handle_event
    df_new = await self._prepare_df(event)
  File "PATH_TO\poptimizer\poptimizer\data\domain\tables\cpi.py", line 32, in _prepare_df
    return await self._gateway()
  File "PATH_TO\poptimizer\poptimizer\data\adapters\gateways\cpi.py", line 80, in __call__
    df = await _load_xlsx(self._session)
  File "PATH_TO\poptimizer\poptimizer\data\adapters\gateways\cpi.py", line 38, in _load_xlsx
    return pd.read_excel(
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\util\_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\io\excel\_base.py", line 364, in read_excel
    io = ExcelFile(io, storage_options=storage_options, engine=engine)
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\io\excel\_base.py", line 1233, in __init__
    self._reader = self._engines[engine](self._io, storage_options=storage_options)
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\io\excel\_openpyxl.py", line 522, in __init__
    super().__init__(filepath_or_buffer, storage_options=storage_options)
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\io\excel\_base.py", line 425, in __init__
    self.book = self.load_workbook(BytesIO(self.handles.handle))
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\io\excel\_openpyxl.py", line 533, in load_workbook
    return load_workbook(
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\openpyxl\reader\excel.py", line 315, in load_workbook
    reader = ExcelReader(filename, read_only, keep_vba,
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\openpyxl\reader\excel.py", line 124, in __init__
    self.archive = _validate_archive(fn)
  File "PATH_TO\anaconda3\envs\finan\lib\site-packages\openpyxl\reader\excel.py", line 96, in _validate_archive
    archive = ZipFile(filename, 'r')
  File "PATH_TO\anaconda3\envs\finan\lib\zipfile.py", line 1257, in __init__
    self._RealGetContents()
  File "PATH_TO\anaconda3\envs\finan\lib\zipfile.py", line 1324, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
WLM1ke commented 2 years ago

Я вчера вернулся наконец к компьютеру и у меня такая же штука произошла. Похоже в очередной раз парсер по инфляции поломался. Временное решение в этой строчке удалить ports.CPI. В ближайшее время починю: https://github.com/WLM1ke/poptimizer/blob/084ac14ca6212a5b14bea5bbc9bb575da077ffb0/poptimizer/data/domain/handlers.py#L61

RomaKoks commented 2 years ago

Понял, спасибо! Кстати, если ошибка происходит при загрузке, то будучи в обработчике try catch - она снова произойдёт. То есть её будет видно.

YoukaiCat commented 2 years ago

Ошибка в парсере инфляции, вроде как, связана с тем, что просто имя файла изменилось на сайте rosstat.gov.ru

--- a/poptimizer/data/adapters/gateways/cpi.py
+++ b/poptimizer/data/adapters/gateways/cpi.py
@@ -10,7 +10,7 @@ from poptimizer.shared import adapters, col

 # Параметры загрузки валидации данных
 # Был адрес URL = "https://rosstat.gov.ru/storage/mediabank/i_ipc_1991-2021(1).xlsx"
-URL = "https://rosstat.gov.ru/storage/mediabank/i_ipc-1991-2021.xlsx"
+URL = "https://rosstat.gov.ru/storage/mediabank/i_ipc_1991-2021.xlsx"
 END_OF_JAN = 31
 PARSING_PARAMETERS = types.MappingProxyType(
     {

Однако, эволюция у меня всё равно не работает.

INFO:EventBus:DateCheckRequired(timestamp=datetime.datetime(2021, 11, 12, 12, 3, 2, 560188))
Traceback (most recent call last):
  File "/home/parsee/poptimizer/./poptimizer/__main__.py", line 37, in <module>
    app(prog_name="poptimizer")
  File "/home/parsee/miniconda3/lib/python3.9/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/parsee/miniconda3/lib/python3.9/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/parsee/poptimizer/./poptimizer/__main__.py", line 12, in evolve
    ev.evolve()
  File "/home/parsee/poptimizer/poptimizer/evolve/evolve.py", line 51, in evolve
    step, current = self._step_setup(step, current)
  File "/home/parsee/poptimizer/poptimizer/evolve/evolve.py", line 84, in _step_setup
    dates = indexes.mcftrr(listing.last_history_date()).loc[self._end :].index
  File "/home/parsee/poptimizer/poptimizer/data/views/indexes.py", line 34, in mcftrr
    df = not_div.index()
  File "/home/parsee/poptimizer/poptimizer/data/views/crop/not_div.py", line 20, in index
    df = viewer.get_df(ports.INDEX, ticker)
  File "/home/parsee/poptimizer/poptimizer/data/app/viewers.py", line 30, in get_df
    return self._loop.run_until_complete(self._query(group, name))
  File "/home/parsee/miniconda3/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/parsee/poptimizer/poptimizer/data/app/viewers.py", line 51, in _query
    raise NoDFError(group, name)
poptimizer.data.app.viewers.NoDFError: ('indexes', 'MCFTRR')
RomaKoks commented 2 years ago

После того как удалил ports.CPI https://github.com/WLM1ke/poptimizer/blob/084ac14ca6212a5b14bea5bbc9bb575da077ffb0/poptimizer/data/domain/handlers.py#L61 получил следующую: 

***2021-11-10: Шаг эволюции — 1***
LLH - (2.0009, 2.6294, 2.6760)
RET - (-0.1670, 0.6095, 1.2661)
Организмов - 115 / Максимум оценок - 9
Доля принятых - 0.62%

Родитель:
LLH — 2.6334: 2.6798, 2.7248, 2.6568, 2.6172, 2.5751, 2.5463
RET — -0.1670: -0.0977, -0.0976, -0.1017, -0.1636, -0.3698, -0.1718
Timer — 0:07:13
Data: {'batch_size': 364.8003048901118, 'history_days': 145.85918648135237, 'ticker_on': 0.7621998239512555, 'day_of_year_on': -0.6076649500389857, 'day_of_period_on': -0.9382465750427933, 'prices_on': 0.8491138898258743, 'dividends_on': 0.7146590707124107, 'turnover_on': 0.2953568169659728, 'average_turnover_on': 0.028678163693356407, 'rvi_on': 0.7548905460393909, 'mcftrr_on': 0.7608859812766444, 'imoex_on': 0.39748660334304553, 'ticker_type_on': -0.39312774183048216, 'usd_on': -0.9319371005496981, 'open_on': -0.36963087425202734, 'high_on': 0.6478021355386165, 'low_on': 0.4755619792138399, 'meogtrr_on': -0.4008841426005453}
Model: {'start_bn': 0.9082809633855027, 'kernels': 5.519242387261828, 'sub_blocks': 1.913408674153375, 'gate_channels': 5.421702136236713, 'residual_channels': 5.742088152681651, 'skip_channels': 5.79683837655868, 'end_channels': 6.778114491569525, 'mixture_size': 2.2639913135882805}
Optimizer: {'betas': 0.9989975363765912, 'eps': 1.2740984374675402e-08, 'weight_decay': 0.041211610525038085}
Scheduler: {'max_lr': 0.002815209322266842, 'epochs': 2.1402683112932857, 'pct_start': 0.300740168046454, 'anneal_strategy': -0.6889762348128361, 'base_momentum': 0.8504079029678564, 'max_momentum': 0.9507583612702956, 'div_factor': 290.3625415175893, 'final_div_factor': 46623.3084230237} 

~~> Test: 100%|██████████| 1/1 [00:05<00:00,  5.67s/it, 2.56480]
RET = -19.71% / MEAN = 41.65% / PLAN = 58.23% / STD = 18.34% / DD = 5.78% / POS = 3 / MAX = 76.26%
Traceback (most recent call last):
  File "PATH_TO\poptimizer\loop.py", line 10, in <module>
    evolve()
  File "PATH_TO\poptimizer\poptimizer\__main__.py", line 12, in evolve
    ev.evolve()
  File "PATH_TO\poptimizer\poptimizer\evolve\evolve.py", line 58, in evolve
    next_, new = self._step(current)
  File "PATH_TO\poptimizer\poptimizer\evolve\evolve.py", line 122, in _step
    if self._eval_organism(hunter) is None:
  File "PATH_TO\poptimizer\poptimizer\evolve\evolve.py", line 186, in _eval_organism
    if self._is_dead(organism):
  File "PATH_TO\poptimizer\poptimizer\evolve\evolve.py", line 197, in _is_dead
    lower, _ = seq.median_conf_bound(ir, config.P_VALUE * 2)
  File "PATH_TO\poptimizer\poptimizer\evolve\seq.py", line 116, in median_conf_bound
    raise SmallSampleError(f"Размер выборки {t} - должен быть не меньше {n}")
poptimizer.evolve.seq.SmallSampleError: Размер выборки 7 - должен быть не меньше 9
Размер выборки 7 - должен быть не меньше 9

Не уверен, что это следствие, но что делать с этой?)

WLM1ke commented 2 years ago

Это проблема перехода между версиями - нужно заменить строчку, где возникла ошибка на: return -np.inf, np.inf