Giskard-AI / cicd

Prototype of CI/CD functionality for Giskard.
Apache License 2.0
2 stars 1 forks source link

Gsk 2559 add tabular classification pipeline #28

Open cy-moi opened 9 months ago

cy-moi commented 9 months ago

Common Problems:

  1. no dataset (blocking)
  2. no config (not blocking)
    • could be solved by using dataset column names and sample predictions
  3. ML console trained models -> unable to process
  4. Pytorch -> Almost no usable models (classification 1, regressions 9 but all the same one bcwarner/audit-icu-gpt2-25_3M)
  5. Most models are scikit-learn with joblib/skops workflows, Keras models are almost all from Keras-io (valid ones less than 10 in total)

Version problems (deserialization): Scikit-learn (skops) Example: python cli.py --loader huggingface --model scikit-learn/Fish-Weight --dataset scikit-learn/Fish

Screenshot 2024-01-17 at 21 51 22

Scikit-learn (joblib) -> XgBoost Example: Almost all titanic models python cli.py --loader huggingface --model vabadeh213/autotrain-titanic-744222727 --dataset phihung/titanic

Screenshot 2024-01-17 at 21 16 18

Keras -> AdamW Example: python cli.py --loader huggingface --model keras-io/tab_transformer --dataset scikit-learn/adult-census-income

Screenshot 2024-01-17 at 21 44 00
linear[bot] commented 9 months ago

GSK-2559 Add customized tabular classification pipeline

Inokinoki commented 9 months ago

Common Problems:

  1. no dataset (blocking)
  2. no config (not blocking)

    • could be solved by using dataset column names and sample predictions
  3. ML console trained models -> unable to process
  4. Pytorch -> Almost no usable models (classification 1, regressions 9 but all the same one bcwarner/audit-icu-gpt2-25_3M)

Version problems (deserialization): Scikit-learn (skops) Example: python cli.py --loader huggingface --model scikit-learn/Fish-Weight --dataset scikit-learn/Fish

Screenshot 2024-01-17 at 21 51 22

Scikit-learn (joblib) -> XgBoost Example: Almost all titanic models python cli.py --loader huggingface --model vabadeh213/autotrain-titanic-744222727 --dataset phihung/titanic

Screenshot 2024-01-17 at 21 16 18

Keras -> AdamW Example: python cli.py --loader huggingface --model keras-io/tab_transformer --dataset scikit-learn/adult-census-income

Screenshot 2024-01-17 at 21 44 00

The last two are apparently deps version issues.

I have also met the first one, no clue.