Workflow to visualize Tpot results

TheodoreGalanos commented 7 years ago

Hello everyone,

I hope this issue hasn't been discussed already. I tried a search through the issues with no luck.

I was wondering if anyone knows a way to quickly visualize the results coming out of Tpot. I feel that performance information across different solvers and across different parameter settings of individual or family of solvers can be extremely useful for all users, and especially new users like me.

I wonder if this can be a feature of Tpot? It can output information about performance of different solvers (perhaps visualizing ranges, colored scatter plots, etc.) and the impact of different parameters on specific solvers in a way that can be easily visualized.

Let me know what you think.

Kind regards, Theodore.

sskarkhanis commented 7 years ago

Hello

just adding my question to this as I thought this question was closest to mine (let me know otherwise, I can raise a separate question),

I've just discovered TPOT this week and having fun learning it so far!

In the description on http://rhiever.github.io/tpot/ ,its mentioned, "TPOT will automate the most tedious part of machine learning by intelligently exploring thousands of possible pipelines to find the best one for your data."

How can I find out which modeling algorithms were considered in the solution?

Thanks!

weixuanfu commented 7 years ago

Hi, Do you mean to check the modeling algorithms / pipelines were evaluated in TPOT optimization process?

You can easily found these pipelines in the dictionary tpot_obj._evaluated_individuals (tpot_obj is a TPOT object) based on the [source codes(https://github.com/rhiever/tpot/blob/master/tpot/base.py#L259-260). The keys of dictionary are pipeline string and the values are the count of operators and pipelines' CV scores.

To be more clear, below it is a small demo:

from tpot import TPOTClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import numpy as np
from deap import creator
from sklearn.model_selection import cross_val_score

# Iris flower classification
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data.astype(np.float64),
    iris.target.astype(np.float64), train_size=0.75, test_size=0.25)

tpot = TPOTClassifier(generations=5, population_size=50, verbosity=2)
tpot.fit(X_train, y_train)
print(tpot.score(X_test, y_test))
# print part of pipeline dictionary
print(dict(list(tpot._evaluated_individuals.items())[0:2]))
# print a pipeline and its values
pipeline_str = list(tpot._evaluated_individuals.keys())[0]
print(pipeline_str)
print(tpot._evaluated_individuals[pipeline_str])
# convert pipeline string to scikit-learn pipeline object
optimized_pipeline = creator.Individual.from_string(pipeline_str, tpot._pset) # deap object
fitted_pipeline = tpot._toolbox.compile(expr=optimized_pipeline ) # scikit-learn pipeline object
# print scikit-learn pipeline object
print(fitted_pipeline)
# Fix random state when the operator allows  (optional) just for get consistent CV score 
tpot._set_param_recursive(fitted_pipeline.steps, 'random_state', 42)
# CV scores from scikit-learn
scores = cross_val_score(fitted_pipeline, X_train, y_train, cv=5, scoring='accuracy', verbose=0)
print(np.mean(scores))
print(tpot._evaluated_individuals[pipeline_str][1])

weixuanfu commented 7 years ago

Update: just fix a small bug in the demo above.

sskarkhanis commented 7 years ago

great! that's what I was looking for... thank you for your response...

zhuangyanbuaa commented 6 years ago

Hi, thanks for your answer. It seems like your code doesn't work now. when I run tpot_obj._evaluated_individuals, error is 'TPOTClassifier' object has no attribute '_evaluated_individuals'

weixuanfu commented 6 years ago

We updated the API since version 0.8. Try ‘tpot_obj.evaluatedindividuals’ instead.

On Dec 7, 2017, at 9:08 AM, yan notifications@github.com<mailto:notifications@github.com> wrote:

evaluated_individuals

sergiolucero commented 6 years ago

how to determine the winning individual? the scores on every value of that dict are all over the place. Can verbosity help?? More insights are needed in the docs. Will dive into the code now.

mikesaclgithubaccount commented 5 years ago

@weixuanfu, is there a way to turn the evaluated_individuals_ key into an sklearn.pipeline without having a fitted TPOTClassifier or TPOTRegressor? I'd like to be record the key strings from evaluated_individuals_ and load them as sklearn.Pipeline objects in sessions after I've fit my TPOTClassifier/Regressor and no longer have access to the initial TPOTClassifier/Regressor that had the evaluated_individuals_.

weixuanfu commented 5 years ago

@miguelehernandez Please try the demo below to convert key strings to Pipeline


from tpot.export_utils import generate_pipeline_code, expr_to_tree

# print a pipeline and its values
pipeline_str = list(tpot.evaluated_individuals_.keys())[0]
print(pipeline_str)
print(tpot.evaluated_individuals_[pipeline_str])
for pipeline_string in sorted(tpot_obj.evaluated_individuals_.keys()):
    # convert pipeline string to scikit-learn pipeline object
    deap_pipeline = creator.Individual.from_string(pipeline_string, tpot_obj._pset)
    sklearn_pipeline = tpot_obj._toolbox.compile(expr=deap_pipeline)
    # print sklearn pipeline string
    sklearn_pipeline_str = generate_pipeline_code(expr_to_tree(individual, tpot_obj._pset), tpot_obj.operators)
    print(sklearn_pipeline_str)```

mikesaclgithubaccount commented 5 years ago

@weixuanfu , thanks so much for the quick reply!

Two follow up questions: 1) Is there an API way to do this that doesn't rely on private methods on the tpot_obj? 2) How does one go about initializing the _pset field when you want to convert the pipeline_string into a sklearn.Pipeline in a session in which the original tpot_obj is not available? For example, say I output the pipeline_string to a file and wanted to read it into a sklearn.Pipeline in another session?

If there is currently no API way to do this, would it be possible for me to contribute something along those lines?

weixuanfu commented 5 years ago

@miguelehernandez So far, there is no API way or a way without using tpot_obj. You're welcome to make contribution to this function!