[FEATURE] Incorporate very-detailed type hints for traversing import hierarchy to enable ML frameworks to be the search space

[SamuelMarks]

Following on from keras-team/keras#18445 have you considered exposing the type information so that you can enumerate:

• categorical All Optimizers, Losses, Layers, Metrics, &etc. available out-of-the-box in JAX, PyTorch, Keras
• continuous What [hyper]parameters are avaiable on each of these

So that one can optimise across these categorical and continuous variables to find, e.g.:

• Best performing—e.g., measured with Metric: AUCROC—Model; or ensemble thereof; on a given dataset
• Best performing Optimizer on a given series of datasets, e.g., for when you're creating a new Optimizer and want to benchmark

Bonus use case

Extending this further, one can expand from just the out-of-the-box solutions and automatically incorporate Models, Optimizers, Losses, &etc. from new research papers into this search-space.

(Somewhat) related work from inside your company:

• https://github.com/google/automl/tree/master/lion
• https://github.com/google/vizier

And from outside I want to reference my own work—publication forthcoming!—at https://github.com/SamuelMarks#multi-ml made possible with my cdd-python

---

Now naturally you can do this from the dynamic level (unlike my solution) or at the static level (actually modify the source code with PEP484 + later type hints). This would make the keras-team/keras#18445 tool create a rather larger git diff if contributed back to the model owners; B-U-T would provide developers with a (sic. IMHO) better experience.

[SamanehSaadat]

Hi @SamuelMarks !

You can already iterate through these. For example for Optimizers you can get the list of optimizers like this:

optimizers = [cls for cls in keras.optimizers.Optimizer.__subclasses__()]

[SamuelMarks]

@SamanehSaadat I know, I was doing that 5+ years ago https://github.com/SamuelMarks/ml-glaucoma/blob/9c8b93/ml_glaucoma/gin_keras.py

The problem is not finding some information dynamically at runtime like that (or with dir); the problem is in knowing:

• Arity
• Argument types
• Argument type ranges (e.g., 8-bit floating point betwixt 0.0 and 1.0)
• Argument defaults

All this information; aside from type ranges; can be found with my newer repo https://github.com/offscale/cdd-python:

import tensorflow as tf

import cdd.class_.parse

# Print this
cdd.class_.parse.class_(
    tf.keras.optimizers.Adam,
    merge_inner_function="__init__"
)

(which internally just uses the builtin ast and inspect libraries)

---

What I keep trying to contribute back to this repository is:

• More precise types
• Type ranges (esp. wrt. numerical and Literal string Unions)

Which would facilitate the creation of a very clean search space that could be optimised across (amongst other advantages, e.g., building GUIs, REST APIs, SQL interfaces to TensorBoard and TFrecords more broadly, &etc. &etc.).

(Still) available and willing to contribute the changes.