tarb / betfair_data

Fast Python Betfair historical data file parser
https://betfair-datascientists.github.io/tutorials/jsonToCsvRevisited/
MIT License
41 stars 5 forks source link

Field Typing and missing attributes #1

Closed varneyo closed 2 years ago

varneyo commented 2 years ago

So apologies if this is incorrect as I dont have any clue about rust with python but the following things I have spotted in the bflw file:

Again apologies if none of this makes sense to how rust works with python. below ive made some changes and added some comments. Would be good to see if anything makes sense in changing:

from datetime import datetime
from typing import Iterator, Optional, Sequence, List, Dict, Any
import betfair_data

class BflwAdapter(Iterator[betfair_data.File]): ...

class MarketDefinitionRunner:
    adjustment_factor: Optional[float] = None
    bsp: Optional[float] = None
    handicap: float
    name: Optional[str] = None
    removal_date: Optional[datetime] = None
    selection_id: int
    sort_priority: int
    status: str

class MarketDefinition:
    bet_delay: int
    betting_type: str
    bsp_market: bool
    bsp_reconciled: bool
    complete: bool
    country_code: str
    cross_matching: bool
    discount_allowed: bool
    each_way_divisor: Optional[str] = None #TODO added missing
    event_id: str
    event_name: Optional[str] = None # changed from eventName
    event_type_id: str
    in_play: bool
    key_line_definitions: Optional[Dict[str, Any]] = None #TODO added create objs? seems to be different from market_book version
    line_interval: Optional[float] = None #TODO lineMinUnit: Optional[float] = None
    line_max_unit: Optional[float] = None #TODO lineMaxUnit: Optional[float] = None
    line_min_unit: Optional[float] = None #TODO lineInterval: Optional[float] = None
    market_base_rate: float
    market_time: datetime
    market_type: str
    name: Optional[str] = None
    number_of_active_runners: int
    number_of_winners: int
    open_date: datetime
    persistence_enabled: bool
    price_ladder_definition: Dict[str, Any] #TODO create an obj? priceLadderDefinition: Dict[str, str]
    race_type: str # raceType: Optional[str] = None
    regulators: List[str]
    runners: List[MarketDefinitionRunner]
    runners_voidable: bool
    settled_time: Optional[datetime] = None
    status: str
    suspend_time: Optional[datetime] = None
    timezone: str
    turn_in_play_enabled: bool
    venue: Optional[str]
    version: int

class RunnerBook:
    adjustment_factor: float
    ex: betfair_data.RunnerBookEX
    handicap: float
    last_price_traded: Optional[float] = None
    removal_date: Optional[datetime] = None
    selection_id: int
    sp: betfair_data.RunnerBookSP
    status: str
    total_matched: float
    matches: List[Any] = []
    orders: List[Any] = []

class MarketBook:
    bet_delay: int
    bsp_reconciled: bool
    complete: bool
    cross_matching: bool
    inplay: bool
    is_market_data_delayed: bool
    last_match_time: datetime
    market_definition: MarketDefinition
    market_id: str
    number_of_active_runners: int
    number_of_runners: int
    number_of_winners: int
    publish_time: datetime
    publish_time_epoch: int
    runners: List[RunnerBook]
    runners_voidable: bool
    status: str
    total_available: float
    total_matched: float
    version: int
    streaming_snap: bool #TODO added for compatability
    streaming_unique_id: int #TODO added for compatability
    streaming_update: Dict[str, Any] #TODO added the last delta message

    _data: Dict[str, Any] #TODO added dict version of market_book object

class File(Iterator[Sequence[MarketBook]]):
    def __init__(self, path: str, bytes: bytes, cumulative_runner_tv: bool = True) -> None: ...
    file_name: str
varneyo commented 2 years ago

Possibly any attribute on the BaseResource also

 def __init__(self, **kwargs):
        self.elapsed_time = kwargs.pop("elapsed_time", None)
        now = datetime.datetime.utcnow()
        self._datetime_created = now
        self._datetime_updated = now
        self._data = kwargs
tarb commented 2 years ago

Thanks for this! Im going to address this in 3 parts.

  1. All the missing, incorrect types have been fixed in 1c7260c, thanks :). A big problem here is that these type (pyi) files are completely independent of the rust source and are created manually. They're only to inform the IDE and help developers and are hand written. This means the process can be a little error prone - errors or missing fields are not picked up by the rust compiler or part of any part of the workflow. I should probably investigate solutions to automate testing them.

  2. The _data field. I'll investigate the best way of doing this, and will create a seperate issue to track that progress.

  3. The streaming/datetime/snap fields. These are fields that pertain to live data. At some point I should investigate an API for deserialising live from the stream, and at that point can determine if/how to incorporate them. For now that its only historical I don't really have any values to put in them so I'll prob just leave them out and leave the API surface area smaller.