Closed rorymac closed 3 years ago
robcarver17
commented
3 years ago That's weird.
If you delete the 'volatility_calculation' line from your backtest .yaml file, does it work? (this will fall back on the defaults.yaml which should be correct).
If you load the system and then type print(system.config.volatility_calculation), what does it show?
rorymac
commented
3 years ago Rob I edited out the lines from the backtest yaml, checked the default.yaml to see that it had a reference to the vol calc (to make sure I had not edited out)
volatility_calculation:
func: "sysquant.estimators.vol.robust_vol_calc"
days: 35
min_periods: 10
vol_abs_min: 0.0000000001
vol_floor: True
floor_min_quant: 0.05
floor_min_periods: 100
floor_days: 500and had the same failure.
full error trace :
rorym@rorym-linux:~/pysystemtrade/sysproduction/linux/scripts$ update_system_backtests
sysproduction.update_system_backtests.update_system_backtests:
None
Arguments:
[]
Which strategy?
0: live_300
1: ALL
Your choice? <RETURN for ALL> 0
/home/rorym/.local/lib/python3.8/site-packages/arctic/_util.py:6: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
from pandas.util.testing import assert_frame_equal
/home/rorym/.local/lib/python3.8/site-packages/arctic/store/_pandas_ndarray_store.py:6: FutureWarning: The Panel class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
from pandas import DataFrame, Series, Panel
Calculating buffers for BOBL
Calculating position method buffer for BOBL
Calculating notional position for BOBL
Calculating diversification multiplier
Calculating instrument weights
Calculating raw instrument weights
Calculating subsystem position for BOBL
FIXME / WARNING: get_default_config_key_value SHOULD NOT BE USED!
Calculating volatility scalar for BOBL
Calculating instrument value vol for BOBL
Calculating instrument currency vol for BOBL
Calculating daily volatility for BOBL
Calculating daily prices for BOBL
Traceback (most recent call last):
File "run.py", line 63, in <module>
func(*args, **kwargs)
File "/home/rorym/pysystemtrade/sysproduction/update_system_backtests.py", line 26, in update_system_backtests
system_backtest_runner.run_strategy_method()
File "/home/rorym/pysystemtrade/syscontrol/strategy_tools.py", line 39, in run_strategy_method
method()
File "/home/rorym/pysystemtrade/sysproduction/strategy_code/run_system_classic.py", line 59, in run_backtest
updated_buffered_positions(data, strategy_name, system)
File "/home/rorym/pysystemtrade/sysproduction/strategy_code/run_system_classic.py", line 138, in updated_buffered_positions
lower_buffer, upper_buffer = get_position_buffers_from_system(
File "/home/rorym/pysystemtrade/sysproduction/strategy_code/run_system_classic.py", line 162, in get_position_buffers_from_system
buffers = system.portfolio.get_buffers_for_position(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 731, in wrapper
ans = system.cache.calc_or_cache(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 589, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "/home/rorym/pysystemtrade/systems/portfolio.py", line 676, in get_buffers_for_position
buffer = self.get_position_method_buffer(instrument_code)
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 731, in wrapper
ans = system.cache.calc_or_cache(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 589, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "/home/rorym/pysystemtrade/systems/portfolio.py", line 586, in get_position_method_buffer
position = abs(self.get_notional_position(instrument_code))
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 731, in wrapper
ans = system.cache.calc_or_cache(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 589, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "/home/rorym/pysystemtrade/systems/portfolio.py", line 542, in get_notional_position
idm = self.get_instrument_diversification_multiplier()
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 708, in wrapper
return func(*args, **kwargs)
File "/home/rorym/pysystemtrade/systems/portfolio.py", line 502, in get_instrument_diversification_multiplier
return self.get_fixed_instrument_diversification_multiplier()
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 731, in wrapper
ans = system.cache.calc_or_cache(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 589, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "/home/rorym/pysystemtrade/systems/portfolio.py", line 490, in get_fixed_instrument_diversification_multiplier
weight_ts = self.get_instrument_weights().index
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 731, in wrapper
ans = system.cache.calc_or_cache(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 589, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "/home/rorym/pysystemtrade/systems/portfolio.py", line 326, in get_instrument_weights
raw_instr_weights = self.get_raw_fixed_instrument_weights()
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 731, in wrapper
ans = system.cache.calc_or_cache(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 589, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "/home/rorym/pysystemtrade/systems/portfolio.py", line 281, in get_raw_fixed_instrument_weights
subsys_ts = [
File "/home/rorym/pysystemtrade/systems/portfolio.py", line 282, in <listcomp>
self.get_subsystem_position(instrument_code).index
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 708, in wrapper
return func(*args, **kwargs)
File "/home/rorym/pysystemtrade/systems/portfolio.py", line 78, in get_subsystem_position
return self.parent.positionSize.get_subsystem_position(instrument_code)
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 731, in wrapper
ans = system.cache.calc_or_cache(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 589, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "/home/rorym/pysystemtrade/systems/positionsizing.py", line 451, in get_subsystem_position
vol_scalar = self.get_volatility_scalar(instrument_code)
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 731, in wrapper
ans = system.cache.calc_or_cache(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 589, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "/home/rorym/pysystemtrade/systems/positionsizing.py", line 404, in get_volatility_scalar
instr_value_vol = self.get_instrument_value_vol(instrument_code)
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 731, in wrapper
ans = system.cache.calc_or_cache(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 589, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "/home/rorym/pysystemtrade/systems/positionsizing.py", line 362, in get_instrument_value_vol
instr_ccy_vol = self.get_instrument_currency_vol(instrument_code)
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 731, in wrapper
ans = system.cache.calc_or_cache(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 589, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "/home/rorym/pysystemtrade/systems/positionsizing.py", line 320, in get_instrument_currency_vol
daily_perc_vol = self.get_price_volatility(instrument_code)
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 731, in wrapper
ans = system.cache.calc_or_cache(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 589, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "/home/rorym/pysystemtrade/systems/positionsizing.py", line 99, in get_price_volatility
daily_perc_vol = system.rawdata.get_daily_percentage_volatility(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 731, in wrapper
ans = system.cache.calc_or_cache(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 589, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "/home/rorym/pysystemtrade/systems/rawdata.py", line 226, in get_daily_percentage_volatility
return_vol = self.daily_returns_volatility(instrument_code)
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 731, in wrapper
ans = system.cache.calc_or_cache(
File "/home/rorym/pysystemtrade/systems/system_cache.py", line 589, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "/home/rorym/pysystemtrade/systems/rawdata.py", line 178, in daily_returns_volatility
vol = volfunction(dailyreturns, **volconfig)
File "/home/rorym/pysystemtrade/syscore/algos.py", line 314, in robust_vol_calc
raise Exception("robust_vol_calc has moved to sysquant.estimators.vol.robust_vol_calc - update your configuration!")
Exception: robust_vol_calc has moved to sysquant.estimators.vol.robust_vol_calc - update your configuration!If you load the system and then type print(system.config.volatility_calculation), what does it show?
rorymac
commented
3 years ago Can you be a little more specific on the following :
If you load the system and then type print(system.config.volatility_calculation), what does it show?
I set up a small python script and tried to run the line to no avail, then tried to point it to the backtest yaml also to no avail. print(private.live_system.live_300_24feb2021.config.volatility_calculation)
but the error may be useful:
NameError: name 'private' is not defined
When I initiated the system from github I used the develop option,
robcarver17
commented
3 years ago from systems.provided.futures_chapter15.basesystem import * futures_system(config = Config(".your.yaml.filename")) print(system.config.volatility_calculation)
(I'm assuming you are using the standard system, but with some different configuration)
rorymac
commented
3 years ago from systems.provided.futures_chapter15.basesystem import *
futures_system(config = Config("private.live_system.live_300_24feb2021.yaml"))
print(system.config.volatility_calculation)output:
rorym@rorym-linux:~/pysystemtrade/private/python_scripts$ python3 print_vol_calc.py
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
Traceback (most recent call last):
File "print_vol_calc.py", line 3, in <module>
print(system.config.volatility_calculation)
NameError: name 'system' is not definedObviously I meant
from systems.provided.futures_chapter15.basesystem import * system = futures_system(config = Config("private.live_system.live_300_24feb2021.yaml")) print(system.config.volatility_calculation)
rorymac
commented
3 years ago Output
rorym@rorym-linux:~/pysystemtrade/private/python_scripts$ python3 print_vol_calc.py
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
{'func': 'syscore.algos.robust_vol_calc', 'days': 35, 'vol_floor': True, 'vol_abs_min': 1e-10, 'min_periods': 10, 'floor_min_quant': 0.05, 'floor_days': 500, 'floor_min_periods': 100}It is pointing to the default.
robcarver17
commented
3 years ago So to be clear: the defaults.yaml file doesn't include ''syscore.algos.robust_vol_calc'? And the "private.live_system.live_300_24feb2021.yaml" file doesn't include it eithier?
robcarver17
commented
3 years ago What does the section volatility_calculation: look like in defaults.yaml? What does it look like in your private yaml file?
rorymac
commented
3 years ago No "private.live_system.live_300_24feb2021.yaml" file does not include it it is edited out.
The default does include it:
volatility_calculation:
func: "sysquant.estimators.vol.robust_vol_calc"
days: 35
min_periods: 10
vol_abs_min: 0.0000000001
vol_floor: True
floor_min_quant: 0.05
floor_min_periods: 100
floor_days: 500In the "private.live_system.live_300_24feb2021.yaml" file
volatility_calculation:
func: "sysquant.estimators.vol.robust_vol_calc"
days: 35though it is currently edited out.
robcarver17
commented
3 years ago from sysdata.config.defaults import * print(get_system_defaults_dict()['volatility_calculation'])
robcarver17
commented
3 years ago from sysdata.config.defaults import * print(get_filename_for_package(DEFAULT_FILENAME))
And check that exact file contains the correct 'volatility_calculation' entry.
rorymac
commented
3 years ago from sysdata.config.defaults import *
print(get_system_defaults_dict()['volatility_calculation'])gives me
{'func': 'sysquant.estimators.vol.robust_vol_calc', 'days': 35, 'min_periods': 10, 'vol_abs_min': 1e-10, 'vol_floor': True, 'floor_min_quant': 0.05, 'floor_min_periods': 100, 'floor_days': 500}
rorymac
commented
3 years ago from sysdata.config.defaults import *
#print(get_filename_for_package(DEFAULT_FILENAME))
print(get_filename_for_package('sysdata.config.defaults.yaml'))rorym@rorym-linux:~/pysystemtrade/private/python_scripts$ python3 print_vol_calc.py
/home/rorym/pysystemtrade/sysdata/config/defaults.yaml
robcarver17
commented
3 years ago I'm stumped.
Try uncommenting the lines in your backtest .yaml line. Then try:
from systems.provided.futures_chapter15.basesystem import * system = futures_system(config = Config("private.live_system.live_300_24feb2021.yaml")) print(system.config.volatility_calculation)
and also try: from systems.provided.futures_chapter15.basesystem import * system = futures_system() print(system.config.volatility_calculation)
Is there any difference in the results?
rorymac
commented
3 years ago Yes there is : BUT it appears only in the order they are displayed...
from systems.provided.futures_chapter15.basesystem import *
system = futures_system(config = Config("private.live_system.live_300_24feb2021.yaml"))
print(system.config.volatility_calculation)rorym@rorym-linux:~/pysystemtrade/private/python_scripts$ python3 print_vol_calc.py
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
{'func': 'syscore.algos.robust_vol_calc', 'days': 35, 'floor_min_periods': 100, 'min_periods': 10, 'floor_days': 500, 'vol_abs_min': 1e-10, 'floor_min_quant': 0.05, 'vol_floor': True}from systems.provided.futures_chapter15.basesystem import *
system = futures_system()
print(system.config.volatility_calculation)rorym@rorym-linux:~/pysystemtrade/private/python_scripts$ python3 print_vol_calc.py
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
No datapaths provided for .csv, will use defaults (may break in production, should be fine in sim)
{'func': 'sysquant.estimators.vol.robust_vol_calc', 'days': 35, 'min_periods': 10, 'vol_abs_min': 1e-10, 'vol_floor': True, 'floor_min_quant': 0.05, 'floor_min_periods': 100, 'floor_days': 500}It is on the test machine and I could start all over and see if I get the same error.
Though I tried it on the spare production server last night and got the same error.
So only have one functional server and getting nervous.. hahaha
robcarver17
commented
3 years ago There is a huge difference! The first one (from "private.live_system.live_300_24feb2021.yaml") is still pointing to the wrong function 'syscore.algos.robust_vol_calc'. The second one, which uses the default.yaml, is correct. 'sysquant.estimators.vol.robust_vol_calc'. So it's definitely your .yaml that is dodgy.
from sysdata.config.defaults import * print(get_filename_for_package("private.live_system.live_300_24feb2021.yaml"))
and check that exact filen contains the correct entry:
volatility_calculation: func: "sysquant.estimators.vol.robust_vol_calc"
and also make sure it doesn't have multiple copies of that entry or something else weird.
So only have one functional server and getting nervous.. hahaha
If it makes you feel any better, my backup server isn't booting at all...
robcarver17
commented
3 years ago If that _exact file__ contains the wrong thing, then deleting and doing a python3 setup.py install on pysystemtrade should do the trick.
rorymac
commented
3 years ago Right so Atom is not perfect, (and it turns out neither an I, quelle horreur) I had left a trail of files behind and not closed them so... it did not update and still had the old one.
Closed and reopened, deleted any references in the "private.live_system.live_300_24feb2021.yaml"
and it went to default and ran.
added in the exact code from default
ran it and it worked.
I am closing this with an honorable mention to the BDFL !
An interesting side effect is the run reduced the V2X position by half...
Put together a fresh setup: Used existing csv files : Futures contracts / multiple / adjusted
Current repo is :
Running :
rorym@rorym-linux:~/pysystemtrade/sysproduction/linux/scripts$ update_system_backtestsYAML Snippet:
How to edit the YAML to prevent the following?
shortened error :
Full error trace: