robcarver17
commented
3 years ago Can I just say first this is very helpful, because I never actually ran the system from a 'cold start' (I backfilled the data from my old system) so it's a great test.
The bad news is I couldn't replicate the problem. I've pasted below the steps I took and added some comments.
If you can do a similar thing, inspecting the stacks and positions after each step, we might be able to work out what's going on.
#Manual position delete in python command line
from sysproduction.data.positions import diagPositions
d = diagPositions()
d.data.db_contract_position.mongo_data._mongo.collection.drop()
d.data.db_strategy_position.mongo_data._mongo.collection.drop()
# back in the shell
rob@boromir:~/pysystemtrade/sysproduction/linux/scripts$ . update_strategy_orders
....
rob@boromir:~/pysystemtrade/sysproduction/linux/scripts$ . interactive_order_stack
sysproduction.interactive_order_stack.interactive_order_stack:
None
Arguments:
[]
0: View
1: Create orders
2: Fills and completions
3: Netting, cancellation and locks
4: Delete and clean
Your choice? <RETURN for EXIT> 0
0: View specific order
1: View instrument order stack
2: View contract order stack
3: View broker order stack (stored local DB)
4: View IB orders and fills
9: View positions
Your choice? <RETURN for Back> 1
INSTRUMENT STACK
(Order ID:30718) Type best for medium_speed_TF_carry AUD, qty [1], fill [0]@ price, None Parent:no parent Children:no_children
.... removed a dozen other orders...
(Order ID:30731) Type best for medium_speed_TF_carry WHEAT, qty [1], fill [0]@ price, None Parent:no parent Children:no_children
0: View specific order
1: View instrument order stack
2: View contract order stack
3: View broker order stack (stored local DB)
4: View IB orders and fills
9: View positions
Your choice? <RETURN for Back> 9
2021-02-02:1040.32 {'type': '', 'component': 'mongoIbBrokerClientIdData'} Locked IB client ID 356
Optimal vs actual
current optimal breaks
medium_speed_TF_carry AEX 0 -0.041/0.041 False
medium_speed_TF_carry PALLAD 0 -0.020/0.020 False
.. removed lots here...
medium_speed_TF_carry AUD 0 0.684/0.947 True
medium_speed_TF_carry OAT 0 1.083/1.345 True
medium_speed_TF_carry WHEAT 0 0.860/1.117 True
Strategy positions
Empty DataFrame
Columns: [strategy_name, instrument_code, position]
Index: []
Contract level positions
Empty DataFrame
Columns: [instrument_code, contract_date, expiry_date, position]
Index: []
(No breaks positions consistent)
Broker positions
Empty DataFrame
Columns: [instrument_code, contract_date, expiry_date, position]
Index: []
(No breaks positions consistent)
0: View specific order
1: View instrument order stack
2: View contract order stack
3: View broker order stack (stored local DB)
4: View IB orders and fills
9: View positions
Your choice? <RETURN for Back>
0: View
1: Create orders
2: Fills and completions
3: Netting, cancellation and locks
4: Delete and clean
Your choice? <RETURN for EXIT> 1
10: Spawn contract orders from instrument orders
11: Create force roll contract orders
12: Create (and try to execute...) IB broker orders
13: Balance trade: Create a series of trades and immediately fill them (not actually executed)
14: Balance instrument trade: Create a trade just at the strategy level and fill (not actually executed)
15: Manual trade: Create a series of trades to be executed
16: Cash FX trade
Your choice? <RETURN for Back> 10
This will create contract orders for any instrument orders that don't have them
Instrument orders:
INSTRUMENT STACK
(Order ID:30718) Type best for medium_speed_TF_carry AUD, qty [1], fill [0]@ price, None Parent:no parent Children:no_children
.... snip...
(Order ID:30731) Type best for medium_speed_TF_carry WHEAT, qty [1], fill [0]@ price, None Parent:no parent Children:no_children
Which instrument order ID <RETURN for All>
Are you sure? (Y/other)Y
Library created, but couldn't enable sharding: no such command: 'enablesharding', bad cmd: '{ enablesharding: "arctic_production", lsid: { id: UUID("fbd85057-e8c6-4864-b3b0-bf7e730ea66a") }, $db: "admin", $readPreference: { mode: "primaryPreferred" } }'. This is OK if you're not 'admin'
Library created, but couldn't enable sharding: no such command: 'enablesharding', bad cmd: '{ enablesharding: "arctic_production", lsid: { id: UUID("a9d80891-e5e5-476c-9a3b-94a4a2c07d34") }, $db: "admin", $readPreference: { mode: "primaryPreferred" } }'. This is OK if you're not 'admin'
2021-02-02:1041.08 {'type': '', 'broker': 'IB', 'clientid': 356, 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'AUD', 'instrument_order_id': 30718} No roll, allocating entire order (Order ID:30718) Type best for medium_speed_TF_carry AUD, qty [1], fill [0]@ price, None Parent:no parent Children:no_children to current contract 20210300
2021-02-02:1041.11 {'type': '', 'broker': 'IB', 'clientid': 356, 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'AUD', 'contract_order_id': '', 'instrument_order_id': 30718} 'Best' order so allocating to original_best
2021-02-02:1041.14 {'type': '', 'broker': 'IB', 'clientid': 356, 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'AUD', 'instrument_order_id': 30718} List of contract orders spawned [(Order ID:no order ID) Type best for medium_speed_TF_carry/AUD/20210300, qty [1], fill [0]@ price, None Parent:30718 Children:no_children]
2021-02-02:1041.16 {'type': '', 'broker': 'IB', 'clientid': 356, 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'AUD', 'instrument_order_id': 30718, 'contract_order_id': ''} Put child order (Order ID:no order ID) Type best for medium_speed_TF_carry/AUD/20210300, qty [1], fill [0]@ price, None Parent:30718 Children:no_children LOCKED on stack with ID 30787 from parent order (Order ID:30718) Type best for medium_speed_TF_carry AUD, qty [1], fill [0]@ price, None Parent:no parent Children:no_children
Library created, but couldn't enable sharding: no such command: 'enablesharding', bad cmd: '{ enablesharding: "arctic_production", lsid: { id: UUID("a323bfab-1ce9-4625-af93-b1c99e4a3298") }, $db: "admin", $readPreference: { mode: "primaryPreferred" } }'. This is OK if you're not 'admin'
Library created, but couldn't enable sharding: no such command: 'enablesharding', bad cmd: '{ enablesharding: "arctic_production", lsid: { id: UUID("1f6369d1-f43e-43e4-8f57-9596847637de") }, $db: "admin", $readPreference: { mode: "primaryPreferred" } }'. This is OK if you're not 'admin'
... snip...
If you are trading manually, you should now view the contract order stack and trade.
Then create manual fills for contract orders
Your choice? <RETURN for Back>
0: View
1: Create orders
2: Fills and completions
3: Netting, cancellation and locks
4: Delete and clean
Your choice? <RETURN for EXIT> 0
0: View specific order
1: View instrument order stack
2: View contract order stack
3: View broker order stack (stored local DB)
4: View IB orders and fills
9: View positions
Your choice? <RETURN for Back> 2
CONTRACT STACK
(Order ID:30787) Type best for medium_speed_TF_carry/AUD/20210300, qty [1], fill [0]@ price, None Parent:30718 Children:no_children
... snip...
(Order ID:30799) Type best for medium_speed_TF_carry/WHEAT/20211200, qty [1], fill [0]@ price, None Parent:30731 Children:no_children
0: View specific order
1: View instrument order stack
2: View contract order stack
3: View broker order stack (stored local DB)
4: View IB orders and fills
9: View positions
Your choice? <RETURN for Back> 1
INSTRUMENT STACK
(Order ID:30718) Type best for medium_speed_TF_carry AUD, qty [1], fill [0]@ price, None Parent:no parent Children:[30787]
... snip
(Order ID:30731) Type best for medium_speed_TF_carry WHEAT, qty [1], fill [0]@ price, None Parent:no parent Children:[30799]
Your choice? <RETURN for Back>
0: View
1: Create orders
2: Fills and completions
3: Netting, cancellation and locks
4: Delete and clean
Your choice? <RETURN for EXIT> 2
20: Manually fill broker or contract order
21: Get broker fills from IB
22: Pass fills upwards from broker to contract order
23: Pass fills upwards from contract to instrument order
24: Handle completed orders
Your choice? <RETURN for Back> 20
Broker stack [1], or Contract stack [2]? <RETURN for Exit> 2
(Order ID:30787) Type best for medium_speed_TF_carry/AUD/20210300, qty [1], fill [0]@ price, None Parent:30718 Children:no_children
... snip....
Enter order ID <RETURN for Cancel> 30787
Order now (Order ID:30787) Type best for medium_speed_TF_carry/AUD/20210300, qty [1], fill [0]@ price, None Parent:30718 Children:no_children
Quantity to fill (must be less than or equal to [1]) <RETURN for default [1]>
Filled price 100
Fill datetime: Enter date and time in format %Y%-%m-%d eg '2020-05-30' OR '%Y-%m-%d %H:%M:%S' eg '2020-05-30 14:04:11' <RETURN for now>
2021-02-02:1045.03 {'type': '', 'component': 'mongoContractOrderStackData', 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'AUD', 'contract_order_id': 30787, 'instrument_order_id': 30718} Changed fill qty from [0] to [1] for order (Order ID:30787) Type best for medium_speed_TF_carry/AUD/20210300, qty [1], fill [0]@ price, None Parent:30718 Children:no_children
### NOTE: THE POSITIONS ARE UPDATED AT THIS POINT SO SHOULD MATCH
2021-02-02:1045.06 {'type': ''} Updated position of AUD/20210300 from 0 to 1; new position in db is 1
2021-02-02:1045.08 {'type': ''} Updated position of AUD/20210300 because of trade (Order ID:30787) Type best for medium_speed_TF_carry/AUD/20210300, qty [1], fill [0]@ price, None Parent:30718 Children:no_children ID:30787 with fills [1]
# THE FILL IS PASSED UP TO THE INSTRUMENT ORDER HERE
2021-02-02:1045.10 {'type': ''} Updated position of medium_speed_TF_carry/AUD from 0 to 1 because of trade (Order ID:30718) Type best for medium_speed_TF_carry AUD, qty [1], fill [0]@ price, None Parent:no parent Children:[30787] 30718
2021-02-02:1045.12 {'type': '', 'component': 'mongoInstrumentOrderStackData', 'strategy_name': 'medium_speed_TF_carry',
'instrument_code': 'AUD', 'instrument_order_id': 30718} Changed fill qty from [0] to [1] for order (Order ID:30718) Type best for medium_speed_TF_carry AUD, qty [1], fill [0]@ price, None Parent:no parent Children:[30787]
Order now (Order ID:30787) Type best for medium_speed_TF_carry/AUD/20210300, qty [1], fill [1]@ price, 100.0 Parent:30718 Children:no_children INACTIVE
## ORDERS ARE NOW COMPLETED AND SHOULD HAVE GONE
If stack process not running, your next job will be to pass fills upwards
20: Manually fill broker or contract order
21: Get broker fills from IB
22: Pass fills upwards from broker to contract order
23: Pass fills upwards from contract to instrument order
24: Handle completed orders
Your choice? <RETURN for Back>
0: View
1: Create orders
2: Fills and completions
3: Netting, cancellation and locks
4: Delete and clean
Your choice? <RETURN for EXIT> 0
0: View specific order
1: View instrument order stack
2: View contract order stack
3: View broker order stack (stored local DB)
4: View IB orders and fills
9: View positions
Your choice? <RETURN for Back> 9
Optimal vs actual
current optimal breaks
medium_speed_TF_carry AEX 0 -0.041/0.041 False
.....
### POSITIONS ARE CORRECT HERE:
Strategy positions
strategy_name instrument_code position
0 medium_speed_TF_carry AUD 1
Contract level positions
instrument_code contract_date expiry_date position
0 AUD 20210315 2021-03-15 00:00:00 1.0
(No breaks positions consistent)
## RUNNING ON A TEST ACCOUNT SO THIS IS TO EXPECTED
Broker positions
Empty DataFrame
Columns: [instrument_code, contract_date, expiry_date, position]
Index: []
BREAKS between broker and DB stored contract positions: [AUD/20210315]
0: View specific order
1: View instrument order stack
2: View contract order stack
3: View broker order stack (stored local DB)
4: View IB orders and fills
9: View positions
Your choice? <RETURN for Back>
0: View
1: Create orders
2: Fills and completions
3: Netting, cancellation and locks
4: Delete and clean
Your choice? <RETURN for EXIT> 2
20: Manually fill broker or contract order
21: Get broker fills from IB
22: Pass fills upwards from broker to contract order
23: Pass fills upwards from contract to instrument order
24: Handle completed orders
Your choice? <RETURN for Back> 23
This will process any fills applied to contract orders and pass them up to instrument orders
CONTRACT STACK
### THE CONTRACT ORDER FOR AUD HAS GONE:
(Order ID:30788) Type best for medium_speed_TF_carry/CAC/20210200, qty [1], fill [0]@ price, None Parent:30719 Children:no_children
.....
Which order ID? <RETURN for for all>
Are you sure? (Y/other)Y
If stack process not running, your next job will be to handle completed orders
20: Manually fill broker or contract order
21: Get broker fills from IB
22: Pass fills upwards from broker to contract order
23: Pass fills upwards from contract to instrument order
24: Handle completed orders
Your choice? <RETURN for Back> 24
This will process any completed orders (all fills present)
INSTRUMENT STACK
### THE INSTRUMENT ORDER FOR AUD HAS GONE:
(Order ID:30719) Type best for medium_speed_TF_carry CAC, qty [1], fill [0]@ price, None Parent:no parent Children:[30788]
Which instrument order ID? <RETURN for All>
Are you sure? (Y/other)Y
20: Manually fill broker or contract order
21: Get broker fills from IB
22: Pass fills upwards from broker to contract order
23: Pass fills upwards from contract to instrument order
24: Handle completed orders
Your choice? <RETURN for Back>
0: View
1: Create orders
2: Fills and completions
3: Netting, cancellation and locks
4: Delete and clean
Your choice? <RETURN for EXIT> 0
0: View specific order
1: View instrument order stack
2: View contract order stack
3: View broker order stack (stored local DB)
4: View IB orders and fills
9: View positions
Your choice? <RETURN for Back> 9
Optimal vs actual
## position now optimal
current optimal breaks
medium_speed_TF_carry AUD 1 0.684/0.947 False
Strategy positions
strategy_name instrument_code position
0 medium_speed_TF_carry AUD 1
Contract level positions
instrument_code contract_date expiry_date position
0 AUD 20210315 2021-03-15 00:00:00 1.0
(No breaks positions consistent)
Broker positions
Empty DataFrame
Columns: [instrument_code, contract_date, expiry_date, position]
Index: []
BREAKS between broker and DB stored contract positions: [AUD/20210315]
0: View specific order
1: View instrument order stack
2: View contract order stack
3: View broker order stack (stored local DB)
4: View IB orders and fills
9: View positions
Your choice? <RETURN for Back>
0: View
1: Create orders
2: Fills and completions
3: Netting, cancellation and locks
4: Delete and clean
Your choice? <RETURN for EXIT> 4
40: Delete entire stack (CAREFUL!)
41: Delete specific order ID (CAREFUL!)
42: End of day process (cancel orders, mark all orders as complete, delete orders)
## RUNNING END OF DAY IN CASE IT DOES SOMETHING WEIRD
Your choice? <RETURN for Back> 42
Will cancel all broker orders, get outstanding fills, mark all orders as complete, update positions, remove everything from stack
Are you sure? (Y/other)Y
2021-02-02:1046.31 {'type': '', 'broker': 'IB', 'clientid': 356} Running safe stack removal
2021-02-02:1046.34 {'type': '', 'broker': 'IB', 'clientid': 356} Trying to cancel all broker orders
2021-02-02:1046.36 {'type': '', 'broker': 'IB', 'clientid': 356} All orders cancelled okay
2021-02-02:1046.38 {'type': '', 'broker': 'IB', 'clientid': 356} Processing fills
2021-02-02:1046.41 {'type': '', 'broker': 'IB', 'clientid': 356} Processing completions
40: Delete entire stack (CAREFUL!)
41: Delete specific order ID (CAREFUL!)
42: End of day process (cancel orders, mark all orders as complete, delete orders)
Your choice? <RETURN for Back>
0: View
1: Create orders
2: Fills and completions
3: Netting, cancellation and locks
4: Delete and clean
Your choice? <RETURN for EXIT> 0
0: View specific order
1: View instrument order stack
2: View contract order stack
3: View broker order stack (stored local DB)
4: View IB orders and fills
9: View positions
Your choice? <RETURN for Back> 9
Optimal vs actual
medium_speed_TF_carry AUD 1 0.684/0.947 False
Strategy positions
strategy_name instrument_code position
0 medium_speed_TF_carry AUD 1
Contract level positions
instrument_code contract_date expiry_date position
0 AUD 20210315 2021-03-15 00:00:00 1.0
(No breaks positions consistent)
Broker positions
Empty DataFrame
Columns: [instrument_code, contract_date, expiry_date, position]
Index: []
BREAKS between broker and DB stored contract positions: [AUD/20210315]
0: View specific order
1: View instrument order stack
2: View contract order stack
3: View broker order stack (stored local DB)
4: View IB orders and fills
9: View positions
Your choice? <RETURN for Back> 1
# DOUBLE CHECK STACKS ARE EMPTY
INSTRUMENT STACK
0: View specific order
1: View instrument order stack
2: View contract order stack
3: View broker order stack (stored local DB)
4: View IB orders and fills
9: View positions
rorymac
I am running the system manually, I do not end up with contract positions. Broker positions balance to Strategy position which balance to optimal positions. What have I missed?
Here is what I did:
I started with a fresh DB, updated from csv files. Manually ran update_system_backtests manually ran update_strategy_orders In interactive_order_stack : 1 Create orders : 10 Spawn contract orders from instrument orders Then : 2 Fills and completions : 20 Manually fill broker or contract order from [2] Contract Stack fill in the details then : 23 Pass fills upwards from contract to instrument order did that for each order Then : 24 Handle completed orders then ran the end of day : 4 Delete and Clean : 42 End of day process.
A balance trade does not work as the instruments have no contract data.
Here is the output : from Interactive_order_stack : 0 View : 9 View positions