Custom user defined mechanism for saving data to alternate storage.

[BennyThadikaran]

@BennyThadikaran a feature request can there be a generic mechanism for saving data that I can extend to write to other datalakes?

Originally posted by @Prady04 in https://github.com/BennyThadikaran/eod2/discussions/54#discussioncomment-9269755

[BennyThadikaran]

I went through the code and below are some of the areas that would require to be implemented.

1. UpdatePendingDeliveryData
2. UpdateNSESymbol
3. MakeAdjustment
4. UpdateIndice
5. RollbackAdjustments
6. RollbackFileUpdates
7. CleanOutdatedFiles

Here is what i thought of:

1. User defines and implements a class that implements the above methods.
2. The class name is mentioned in user.json
3. During update sync, EOD2 will make calls to the class methods passing the necessary parameters.

You wont have to implement every method. EOD2 will check if the method exists and is callable before making the call.

This requires very minimal changes to code. Files are still saved to local filesystem. You can implement the class however you like and for any purpose.

Let me know what you think.

[BennyThadikaran]

@prady04 I will start work on this tomorrow. Will create a separate branch to test it a few days in case any changes are required.

[BennyThadikaran]

I have updated the repo with a new branch feat/hooks.

At the bottom is the list of function hooks with their signatures. You can define them in a python file using either function or using Classes (Whichever you prefer). All functions/methods are optional and only called if defined.

You can save the Python file anywhere on your filesystem.

In your user.json, define INIT_HOOK key with the full path to your Python file. If your hooks are defined in a class, add the class name separated by | like below. The class should not take any arguments during initialization.

I have not merged it yet. Will wait a few days to fix any bugs and make any required changes.

The updateNseSymbol and updateIndice can be used to gather data and make a bulk commit when on_complete is called. This is ideal for performance and in case error occurs.

makeAdjustment is safe to execute since its called only after all adjustments were successful.

{
    "INIT_HOOK": "~/Documents/python/eod2/src/data/hooks.py|MyCustomClass",
}

def updatePendingDeliveryData(delivery_bhav: pd.DataFrame, date: datetime):

def updateNseSymbol(
    date: datetime,
    sym: str,
    open: float,
    high: float,
    low: float,
    close: float,
    volume: int,
    total_trades: Union[int, str],
    qty_per_trade: Union[float, str],
    delivery_qty: Union[int, str],
):

def makeAdjustment(date: datetime, sym_list: List[Tuple[str, float]]):
    """sym_list is a list of tuples

    Each Tuple contains the symbol name and adjustment factor.

    Formula to apply adjustment

    adjusted_price = price / adjustment_factor

    To Round to nearest 0.05: 

    round(adjusted_price / 0.05) * 0.05
    """

def updateIndice(
    date: datetime,
    sym: str,
    open: float,
    high: float,
    low: float,
    close: float,
    volume: int,
):

def cleanOutDated(sym_list: List[str]):
    """
    sym_list: A list of symbols to be removed
    """

def on_complete():
    """
    Called after all symbols updated and adjustments complete
    """

def on_error():
    """
    Called if error occurs and only after rollback is completed
    """

[BennyThadikaran]

Tested on Windows 10. I have replaced the Colon : with Pipe char | when specifying a Python class in INIT_HOOKS.

The : being a special character in Windows filepaths. The previous post has been updated with changes.

[BennyThadikaran]

I have merged the feature into main branch. Closing this issue for now. I will update the WIKI with details of new feature.