'frompackages' directive functionality seems to be broken when using inheritance

Community discussion:

https://community.backtrader.com/topic/2661/frompackages-directive-functionality-seems-to-be-broken-when-using-inheritance

Background:

'frompackages' directive was added to backtrader back in 2017 (starting from release 1.9.30.x). It allows specifying the external packages to be imported only during the instantiation of the class (usually indicator). It comes very handily during the optimizations, reducing the serialization size of the objects. More on this here

Usage example:

class MyIndicator(bt.Indicator):
    frompackages = (('pandas', 'SomeFunction'),)
    lines = ('myline',)
    params = (
        ('period', 50),
    )

    def next(self):
        print('mylines[0]:', SomeFunction(self.lines.myline[0]))

Here the SomeFunction will be imported from pandas package during the instantiation of MyIndicator and not earlier.

Testcase:

In the same article, it was also claimed that "Both packages and frompackages support (multiple) inheritance". However, it seems to be not the case. Here a short test case:

import os
import backtrader as bt

class HurstExponentEx(bt.indicators.HurstExponent):
    def __init__(self):
        super(HurstExponentEx, self).__init__()

    def next(self):
        super(HurstExponentEx, self).next()
        print('test')

class TheStrategy(bt.Strategy):
    def __init__(self):
        self.hurst = HurstExponentEx(self.data, lag_start=10,lag_end=500)

    def next(self):
        print('next')

def runstrat():
    cerebro = bt.Cerebro()
    cerebro.broker.set_cash(1000000)
    data_path = os.path.join(bt.__file__, '../../datas/yhoo-1996-2014.txt')
    data0 = bt.feeds.YahooFinanceCSVData(dataname=data_path)
    cerebro.adddata(data0)
    cerebro.addstrategy(TheStrategy)
    cerebro.run()
    cerebro.plot()

if __name__ == '__main__':
    runstrat()

where the HustExponent class is defined in backtrader as:

class HurstExponent(PeriodN):
    frompackages = (
        ('numpy', ('asarray', 'log10', 'polyfit', 'sqrt', 'std', 'subtract')),
    )
    ...

Unexpected behavior:

trying to run it (using python 3.6 in my case) will produce:

Traceback (most recent call last):
  File "test_frompackage.py", line 42, in <module>
    runstrat()
  File "test_frompackage.py", line 38, in runstrat
    cerebro.run()
  File "W:\backtrader\backtrader\cerebro.py", line 1182, in run
    runstrat = self.runstrategies(iterstrat)
  File "W:\backtrader\backtrader\cerebro.py", line 1275, in runstrategies
    strat = stratcls(*sargs, **skwargs)
  File "W:\backtrader\backtrader\metabase.py", line 88, in __call__
    _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs)
  File "W:\backtrader\backtrader\metabase.py", line 78, in doinit
    _obj.__init__(*args, **kwargs)
  File "test_frompackage.py", line 17, in __init__
    lag_end=500)
  File "W:\backtrader\backtrader\indicator.py", line 53, in __call__
    return super(MetaIndicator, cls).__call__(*args, **kwargs)
  File "W:\backtrader\backtrader\metabase.py", line 88, in __call__
    _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs)
  File "W:\backtrader\backtrader\metabase.py", line 78, in doinit
    _obj.__init__(*args, **kwargs)
  File "test_frompackage.py", line 6, in __init__
    super(HurstExponentEx, self).__init__()
  File "W:\backtrader\backtrader\indicators\hurst.py", line 82, in __init__
    self.lags = asarray(range(lag_start, lag_end))
NameError: name 'asarray' is not defined

as could be seen asarray is a method that should have been imported from numpy package upon instantiation of HurstExponent class.

If we will directly use the HurstExponent class instead of our HurstExponentEx (which inherits from HurstExponent) - everything will work just fine.

Analysis - TL;DR

Exploring this is a little bit exposes the problem with the implementation of frompackages inside backtrader.

The magic code responsible for 'frompackages' directive handling could be found in backtrader\metabase.py file inside the MetaParams.__new__ and MetaParams.donew methods. Here the 'frompackages' directive is first examined recursively (in __new__ method ) and the appropriate packages are imported using __import__ function ( in donew method)

The problem is with the following code inside the donew method:

    def donew(cls, *args, **kwargs):
        clsmod = sys.modules[cls.__module__]
        .
        . <removed for clarity>
        .
        # import from specified packages - the 2nd part is a string or iterable
        for p, frompackage in cls.frompackages:
            if isinstance(frompackage, string_types):
                frompackage = (frompackage,)  # make it a tuple

            for fp in frompackage:
                if isinstance(fp, (tuple, list)):
                    fp, falias = fp
                else:
                    fp, falias = fp, fp  # assumed is string

                # complain "not string" without fp (unicode vs bytes)
                pmod = __import__(p, fromlist=[str(fp)])
                pattr = getattr(pmod, fp)
                setattr(clsmod, falias, pattr)

The cls parameter to this function is the class that needs to be instantiated. In our case, this is our inherited class HustExponentEx.

So the clsmod variable will contain the module of our class - obviously the file that HurstExponentEx was defined in.

The problem is with the last line of the above code:

setattr(clsmod, falias, pattr)

Here the setattr will introduce the imported names to the module - our module with inherited class - not the module the original base class HurstExponent is defined in.

And it is a problem!

Once the HurstExponent class will start executing and calling the supposedly imported functions - those will be looked in the module the HurstExponent class is defined in - and will not be found, since those names are introduced in the module of our inherited class instead!

FIX

The fix seems to be obvious. Introduce the imported names to the original base class module.

backtrader2 / backtrader