Closed JimLundin closed 1 month ago
Hi! 👋 Thanks for the contribution. Feel free to add additional typing that you think is required (e.g. to the pyspark transform)
Added the pyspark typing to transform_df! There are more typing improvements that I think could be made to the decorators. If you would like I can continue adding to this PR or start a new one.
Feel free to add further typing - let me know when you think it’s ready to be merged. Thanks!
PR looks good now! I have focused on fixing the developer-facing type hints for the transform decorators. You are welcome to merge or give feedback on the changes. When I have more time, I could help set up a github workflow for type checking and add more "internal" type hints. If that fits into the philosophy of this library.
feel free to contribute additional typing PRs where you think it makes sense.
Summary
Hello!
We are utilising foundry-dev-tools for some local development, primarily for the added type hints when using transforms.
We use the "strict=True" option for mypy type checking during our development. This setting disallows unspecified Callable types as an error when checking types. I added more specific gradual types to the "transformation_polars" and "transformation_pandas" decorators so that mypy is satisfied.
I suspect that the same type hints could be added for the pyspark transforms aswell. I can add that to this PR if that is desirable.
In the process, I also added the "Input" type to the **kwargs in the decorator signature. If required I could split that into a separate PR.
Best regards, Jim
Checklist