Prepare for next releases of Numba

[hmgaudecker]

Version 0.0.60 gives a NumbaDepreciationWarning for the translog function because it falls back to object mode (=no speed improvement), they will disable that in the future.

Solution: Either make compliant with Numba's supported features of Python or remove the decorator.

NB: Most of the stuff below seems repetitive, but I did not check super-carefully whether some part is not, so I rather dump the whole output...

/home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py:116: NumbaWarning: 
Compilation is falling back to object mode WITH looplifting enabled because Function "translog" failed type inference due to: Invalid use of Function(<built-in function getitem>) with argument(s) of type(s): (tuple(int64 x 5), slice<a:b>)
 * parameterized
In definition 0:
    All templates rejected with literals.
In definition 1:
    All templates rejected without literals.
In definition 2:
    All templates rejected with literals.
In definition 3:
    All templates rejected without literals.
In definition 4:
    All templates rejected with literals.
In definition 5:
    All templates rejected without literals.
In definition 6:
    All templates rejected with literals.
In definition 7:
    All templates rejected without literals.
In definition 8:
    All templates rejected with literals.
In definition 9:
    All templates rejected without literals.
In definition 10:
    All templates rejected with literals.
In definition 11:
    All templates rejected without literals.
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: typing of intrinsic-call at /home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py (136)

File "../../../../../miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py", line 136:
def translog(sigma_points, coeffs, included_positions):
    <source elided>
            # add the interaction terms
            for pos2 in included_positions[p:]:
            ^

  @jit
/home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py:116: NumbaWarning: 
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "translog" failed type inference due to: cannot determine Numba type of <class 'numba.dispatcher.LiftedLoop'>

File "../../../../../miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py", line 125:
def translog(sigma_points, coeffs, included_positions):
    <source elided>
    nr_included = len(included_positions)
    for i in range(long_side):
    ^

  @jit
/home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/numba/compiler.py:742: NumbaWarning: Function "translog" was compiled in object mode without forceobj=True, but has lifted loops.

File "../../../../../miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py", line 117:
@jit
def translog(sigma_points, coeffs, included_positions):
^

  self.func_ir.loc))
/home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/numba/compiler.py:751: NumbaDeprecationWarning: 
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

File "../../../../../miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py", line 117:
@jit
def translog(sigma_points, coeffs, included_positions):
^

  warnings.warn(errors.NumbaDeprecationWarning(msg, self.func_ir.loc))
/home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py:116: NumbaWarning: 
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "translog" failed type inference due to: Invalid use of Function(<built-in function getitem>) with argument(s) of type(s): (tuple(int64 x 5), slice<a:b>)
 * parameterized
In definition 0:
    All templates rejected with literals.
In definition 1:
    All templates rejected without literals.
In definition 2:
    All templates rejected with literals.
In definition 3:
    All templates rejected without literals.
In definition 4:
    All templates rejected with literals.
In definition 5:
    All templates rejected without literals.
In definition 6:
    All templates rejected with literals.
In definition 7:
    All templates rejected without literals.
In definition 8:
    All templates rejected with literals.
In definition 9:
    All templates rejected without literals.
In definition 10:
    All templates rejected with literals.
In definition 11:
    All templates rejected without literals.
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: typing of intrinsic-call at /home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py (136)

File "../../../../../miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py", line 136:
def translog(sigma_points, coeffs, included_positions):
    <source elided>
            # add the interaction terms
            for pos2 in included_positions[p:]:
            ^

  @jit
/home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/numba/compiler.py:742: NumbaWarning: Function "translog" was compiled in object mode without forceobj=True.

File "../../../../../miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py", line 125:
def translog(sigma_points, coeffs, included_positions):
    <source elided>
    nr_included = len(included_positions)
    for i in range(long_side):
    ^

  self.func_ir.loc))
/home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/numba/compiler.py:751: NumbaDeprecationWarning: 
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

File "../../../../../miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py", line 125:
def translog(sigma_points, coeffs, included_positions):
    <source elided>
    nr_included = len(included_positions)
    for i in range(long_side):
    ^

  warnings.warn(errors.NumbaDeprecationWarning(msg, self.func_ir.loc))
/home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py:116: NumbaWarning: 
Compilation is falling back to object mode WITH looplifting enabled because Function "translog" failed type inference due to: Invalid use of Function(<built-in function getitem>) with argument(s) of type(s): (tuple(int64 x 3), slice<a:b>)
 * parameterized
In definition 0:
    All templates rejected with literals.
In definition 1:
    All templates rejected without literals.
In definition 2:
    All templates rejected with literals.
In definition 3:
    All templates rejected without literals.
In definition 4:
    All templates rejected with literals.
In definition 5:
    All templates rejected without literals.
In definition 6:
    All templates rejected with literals.
In definition 7:
    All templates rejected without literals.
In definition 8:
    All templates rejected with literals.
In definition 9:
    All templates rejected without literals.
In definition 10:
    All templates rejected with literals.
In definition 11:
    All templates rejected without literals.
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: typing of intrinsic-call at /home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py (136)

File "../../../../../miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py", line 136:
def translog(sigma_points, coeffs, included_positions):
    <source elided>
            # add the interaction terms
            for pos2 in included_positions[p:]:
            ^

  @jit
/home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py:116: NumbaWarning: 
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "translog" failed type inference due to: cannot determine Numba type of <class 'numba.dispatcher.LiftedLoop'>

File "../../../../../miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py", line 125:
def translog(sigma_points, coeffs, included_positions):
    <source elided>
    nr_included = len(included_positions)
    for i in range(long_side):
    ^

  @jit
/home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/numba/compiler.py:742: NumbaWarning: Function "translog" was compiled in object mode without forceobj=True, but has lifted loops.

File "../../../../../miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py", line 117:
@jit
def translog(sigma_points, coeffs, included_positions):
^

  self.func_ir.loc))
/home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/numba/compiler.py:751: NumbaDeprecationWarning: 
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

File "../../../../../miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py", line 117:
@jit
def translog(sigma_points, coeffs, included_positions):
^

  warnings.warn(errors.NumbaDeprecationWarning(msg, self.func_ir.loc))
/home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py:116: NumbaWarning: 
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "translog" failed type inference due to: Invalid use of Function(<built-in function getitem>) with argument(s) of type(s): (tuple(int64 x 3), slice<a:b>)
 * parameterized
In definition 0:
    All templates rejected with literals.
In definition 1:
    All templates rejected without literals.
In definition 2:
    All templates rejected with literals.
In definition 3:
    All templates rejected without literals.
In definition 4:
    All templates rejected with literals.
In definition 5:
    All templates rejected without literals.
In definition 6:
    All templates rejected with literals.
In definition 7:
    All templates rejected without literals.
In definition 8:
    All templates rejected with literals.
In definition 9:
    All templates rejected without literals.
In definition 10:
    All templates rejected with literals.
In definition 11:
    All templates rejected without literals.
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: typing of intrinsic-call at /home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py (136)

File "../../../../../miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py", line 136:
def translog(sigma_points, coeffs, included_positions):
    <source elided>
            # add the interaction terms
            for pos2 in included_positions[p:]:
            ^

  @jit
/home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/numba/compiler.py:742: NumbaWarning: Function "translog" was compiled in object mode without forceobj=True.

File "../../../../../miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py", line 125:
def translog(sigma_points, coeffs, included_positions):
    <source elided>
    nr_included = len(included_positions)
    for i in range(long_side):
    ^

  self.func_ir.loc))
/home/wherever/miniconda/envs/myenv/lib/python3.6/site-packages/numba/compiler.py:751: NumbaDeprecationWarning: 
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

File "../../../../../miniconda/envs/myenv/lib/python3.6/site-packages/skillmodels/model_functions/transition_functions.py", line 125:
def translog(sigma_points, coeffs, included_positions):
    <source elided>
    nr_included = len(included_positions)
    for i in range(long_side):
    ^

  warnings.warn(errors.NumbaDeprecationWarning(msg, self.func_ir.loc))

[hmgaudecker]

~Hmmm, this seems to be related to fixing some parameters to values other than the start value.~

~In the first shot, which yielded the warning, I had not bothered about the start values of 0.1 for some constants fixed to zero in an AR(1) equation.~

~Doing so also seems to have removed the above.~

Nope, the reason was just that I had not restarted the Kernel and Numba did not need to recompile.

[janosg]

Yes, the warning when you fix a parameter to a different value than it's start value is just a harmless estimagic warning.

I am very sure that translog did compile in nopython mode, so it must somehow be related to your specific inputs or the changed data structures.

Could you pickle the output of SkillModel.likelihood_arguments_dict and send them to me? Since this will contain parts of your dataset you should not post it publicly and ideally use a model instance generated from a small subset of your data.

[hmgaudecker]

Thanks. Can't do that for privacy reasons. You will need to tell me what I should be looking for, I am afraid.

[janosg]

I'm happy if you do the debugging for me :-). I don't really know what I would be looking for. Probably some empty lists / tuples /arrays or slices thereof, that only occur in your constellation.

The inputs for the translog function will be in likelihood_arguments_dict["predict_args"]["transform_sigma_point_args"]["transition_argument_dicts"] (I know that's not ideal ...) and this does not contain raw data or enough information to reconstruct data even if someone was willing to write the inverse of a Kalman Filter to get at your data.

[hmgaudecker]

Okay, that is indeed just a bunch of zeros, see below.

~The problem is very likely the last one, where the coeffs array is empty. I probably put in a restriction too many in my attempt to re-create the AR(1) specification. Will have to check later.~

~If my assertion is correct, it seems like this would be worth a testcase and a graceful error message now that more responsibility for restrictions is with the user.~

I just realise this happens in every element. I have five factos (constant, translog, translog, AR(1), AR(1)) and four periods So I guess this sorts the factors by their names and the constant one is last. Then this looks right.

[
    [
        {"coeffs": array([0.0, 0.0, 0.0]), "included_positions": (0, 4)},
        {"coeffs": array([0.0, 0.0, 0.0]), "included_positions": (1, 4)},
        {
            "coeffs": array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]),
            "included_positions": (0, 1, 2, 3, 4),
        },
        {"coeffs": array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]), "included_positions": (1, 3, 4)},
        {"coeffs": array([], dtype=float64), "included_positions": (4,)},
    ],
    [
        {"coeffs": array([0.0, 0.0, 0.0]), "included_positions": (0, 4)},
        {"coeffs": array([0.0, 0.0, 0.0]), "included_positions": (1, 4)},
        {
            "coeffs": array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]),
            "included_positions": (0, 1, 2, 3, 4),
        },
        {"coeffs": array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]), "included_positions": (1, 3, 4)},
        {"coeffs": array([], dtype=float64), "included_positions": (4,)},
    ],
    [
        {"coeffs": array([0.0, 0.0, 0.0]), "included_positions": (0, 4)},
        {"coeffs": array([0.0, 0.0, 0.0]), "included_positions": (1, 4)},
        {
            "coeffs": array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]),
            "included_positions": (0, 1, 2, 3, 4),
        },
        {"coeffs": array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]), "included_positions": (1, 3, 4)},
        {"coeffs": array([], dtype=float64), "included_positions": (4,)},
    ],
    [
        {"coeffs": array([0.0, 0.0, 0.0]), "included_positions": (0, 4)},
        {"coeffs": array([0.0, 0.0, 0.0]), "included_positions": (1, 4)},
        {
            "coeffs": array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]),
            "included_positions": (0, 1, 2, 3, 4),
        },
        {"coeffs": array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]), "included_positions": (1, 3, 4)},
        {"coeffs": array([], dtype=float64), "included_positions": (4,)},
    ],
]

[janosg]

This looks like a bug, not a user error. The coeffs array is constructed based on the number of included positions.

Just to locate the problem: If I remember correctly you got the error in a translog function. Are you trying to express the AR1 Process as translog? Or as a linear function (which would be easier and probably faster)

[hmgaudecker]

AR(1) is linear + restrictions.

[janosg]

Yes, the only problem seems to be that included_positions is a tuple, not an array. Will be fixed soon.