[Plugin] Numpy dtypes integration

[cosmicBboy]

Currently, only numpy arrays are supported by the numpy type extension. It would be useful to also support scalar data types as well.

So today, this code will not work:

import numpy as np
from flytekit import task, workflow

@task
def my_task() -> float:
    x = np.array([1,2,3])
    return x.mean()

@workflow
def wf():
    my_task()

wf()

Error:

TypeError: Failed to convert return value for var o0 for function my_task with error <class 'flytekit.core.type_engine.TypeTransformerFailedError'>: Expected value of type <class 'float'> but got type <class 'numpy.float64'>

Problem: flytekit doesn't know how to handle numpy.<dtype> scalar types. The code above will show up anywhere where a user wants to aggregate some values using numpy operations (including sklearn, and probably a bunch of others)

Of course the user can always do float(x.mean()) to convert it to a normal float, but then this kills UX because now we're forcing users to modify otherwise functioning code into some flytekit-nitpicky syntax.

Potential Solutions

1. Create a set of type transformers for numpy scalar dtypes
2. Modify existing primitive type transformers (see here) to recognize numpy types.

Resources

• numpy scalar types: https://numpy.org/doc/stable/reference/arrays.scalars.html

[SmritiSatyanV]

Could I work on this? @cosmicBboy

[aryamans29002]

can i work on this @cosmicBboy

[techytushar]

Hey @cosmicBboy , I have created a PR to fix this https://github.com/flyteorg/flytekit/pull/1219 Please can you check if this is the right way to solve this, so I can go ahead and add the tests as well

[P3rcy-8685]

I would like to work on this @cosmicBboy

[samhita-alla]

Since @techytushar created a PR already, I assigned the issue to Tushar.

[github-actions[bot]]

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable. Thank you for your contribution and understanding! 🙏

[Monis-Ahmed-Rizvi]

I’d like to take on this issue if no one else is working on it.