search

PAIR-code / lit

The Learning Interpretability Tool: Interactively analyze ML models to understand their behavior in an extensible and framework agnostic interface.
https://pair-code.github.io/lit
Apache License 2.0
3.47k stars 353 forks source link

Shape Mismatch Error in umap calculation when entering a custom datapoint #15

Closed rwikdutta closed 4 years ago

rwikdutta commented 4 years ago

Hi, I modified the quickstart_sst_demo.py example file to allow it to run already fine-tuned models from huggingface without the need to first train it. I had loaded this model https://huggingface.co/sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english into the tool and was using it. It was working fine, until I entered a custom sentence in Datapoint Editor ( i.e. not in the loaded validation set ) and clicked on Make New Datapoint and got the following error:

E0822 15:21:16.088476 140224925210432 wsgi_app.py:210] Uncaught error: Incompatible dimension for X and Y matrices: X.shape[1] == 2 while Y.shape[1] == 128

I had earlier tried this model also https://huggingface.co/textattack/bert-base-uncased-SST-2 and that too gave the same error.

I went back and checked the same thing when using the default script ( where google/bert_uncased_L-2_H-128_A-2 is first loaded, fine tuned for 3 epochs and then this model is used in LIT ), this issue didn't take place.

Logs while loading the distilbert model:

2020-08-22 15:19:36.691398: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-08-22 15:19:36.691460: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
I0822 15:19:38.790888 140224925210432 quickstart_sst_demo_huggingface.py:57] Working directory: sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english
I0822 15:19:38.791229 140224925210432 quickstart_sst_demo_huggingface.py:61] train=False, so no finetuning will be done
I0822 15:19:38.870157 140224925210432 configuration_utils.py:265] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english/config.json from cache at /root/.cache/torch/transformers/8ea67766c9c6c502b1e44635404952d1a5685e296fe18144020458c9b07710c5.2677af367f3d00baa869333b4aa545410fb85939ec3a67ddade35baade7f93fe
I0822 15:19:38.870624 140224925210432 configuration_utils.py:301] Model config DistilBertConfig {
  "activation": "gelu",
  "architectures": [
    "DistilBertForSequenceClassification"
  ],
  "attention_dropout": 0.1,
  "dim": 2,
  "dropout": 0.1,
  "finetuning_task": "sst-2",
  "hidden_dim": 2,
  "id2label": {
    "0": "NEGATIVE",
    "1": "POSITIVE"
  },
  "initializer_range": 0.02,
  "label2id": {
    "NEGATIVE": 0,
    "POSITIVE": 1
  },
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 2,
  "n_layers": 2,
  "output_past": true,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "vocab_size": 30522
}

I0822 15:19:38.870761 140224925210432 tokenization_utils.py:938] Model name 'sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english' not found in model shortcut name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad, distilbert-base-cased, distilbert-base-cased-distilled-squad, distilbert-base-german-cased, distilbert-base-multilingual-cased). Assuming 'sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english' is a path, a model identifier, or url to a directory containing tokenizer files.
I0822 15:19:39.196162 140224925210432 tokenization_utils.py:1022] loading file https://s3.amazonaws.com/models.huggingface.co/bert/sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english/vocab.txt from cache at /root/.cache/torch/transformers/45a97749e7cd555a7d4a5068597027de77253a260bb784f256b8467034afcbae.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
I0822 15:19:39.196318 140224925210432 tokenization_utils.py:1022] loading file https://s3.amazonaws.com/models.huggingface.co/bert/sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english/added_tokens.json from cache at None
I0822 15:19:39.196394 140224925210432 tokenization_utils.py:1022] loading file https://s3.amazonaws.com/models.huggingface.co/bert/sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english/special_tokens_map.json from cache at /root/.cache/torch/transformers/11de36abdf85710b886b11de7c2a654913de2a5219d1d2c8909e39529244885e.275045728fbf41c11d3dae08b8742c054377e18d92cc7b72b6351152a99b64e4
I0822 15:19:39.196466 140224925210432 tokenization_utils.py:1022] loading file https://s3.amazonaws.com/models.huggingface.co/bert/sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english/tokenizer_config.json from cache at /root/.cache/torch/transformers/a33dd44cc977304f8be92eeaf59c61314eec22b13a4c571a559f2b4d23d66e93.3889713104075cfee9e96090bcdd0dc753733b3db9da20d1dd8b2cd1030536a2
I0822 15:19:39.304257 140224925210432 configuration_utils.py:265] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english/config.json from cache at /root/.cache/torch/transformers/8ea67766c9c6c502b1e44635404952d1a5685e296fe18144020458c9b07710c5.2677af367f3d00baa869333b4aa545410fb85939ec3a67ddade35baade7f93fe
I0822 15:19:39.304666 140224925210432 configuration_utils.py:301] Model config DistilBertConfig {
  "activation": "gelu",
  "architectures": [
    "DistilBertForSequenceClassification"
  ],
  "attention_dropout": 0.1,
  "dim": 2,
  "dropout": 0.1,
  "finetuning_task": "sst-2",
  "hidden_dim": 2,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 2,
  "n_layers": 2,
  "output_attentions": true,
  "output_hidden_states": true,
  "output_past": true,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "vocab_size": 30522
}

I0822 15:19:39.559951 140224925210432 modeling_tf_utils.py:384] loading weights file https://cdn.huggingface.co/sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english/tf_model.h5 from cache at /root/.cache/torch/transformers/c151044f03ea62cb9b121f822633cfe511ec33c640f82a5c36643935d3aa56c3.b4173921ecda9322480ca95ae50a7de71bc1ef95e0ab546577648ccc3d3a6eb9.h5
2020-08-22 15:19:39.580763: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-08-22 15:19:39.580810: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
2020-08-22 15:19:39.580853: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (9474283738e4): /proc/driver/nvidia/version does not exist
2020-08-22 15:19:39.581088: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-08-22 15:19:39.588133: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300050000 Hz
2020-08-22 15:19:39.588746: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5641253f24b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-22 15:19:39.588784: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-08-22 15:19:39.603479: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
I0822 15:19:39.732185 140224925210432 modeling_tf_utils.py:422] Layers of TFDistilBertForSequenceClassification not initialized from pretrained model: ['dropout_7']
I0822 15:19:39.732364 140224925210432 modeling_tf_utils.py:426] Layers from pretrained model not used in TFDistilBertForSequenceClassification: ['dropout_484']
2020-08-22 15:19:39.831450: W tensorflow/core/platform/cloud/google_auth_provider.cc:184] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with "Not found: Could not locate the credentials file.". Retrieving token from GCE failed with "Failed precondition: Error executing an HTTP request: libcurl code 6 meaning 'Couldn't resolve host name', error details: Couldn't resolve host 'metadata'".
I0822 15:19:40.051693 140224925210432 dataset_info.py:358] Load dataset info from /root/tensorflow_datasets/glue/sst2/1.0.0
I0822 15:19:40.053645 140224925210432 dataset_builder.py:288] Reusing dataset glue (/root/tensorflow_datasets/glue/sst2/1.0.0)
I0822 15:19:40.053833 140224925210432 dataset_builder.py:500] Constructing tf.data.Dataset for split validation, from /root/tensorflow_datasets/glue/sst2/1.0.0
I0822 15:19:40.356983 140224925210432 dev_server.py:79] 
 (    (           
 )\ ) )\ )  *   ) 
(()/((()/(` )  /( 
 /(_))/(_))( )(_))
(_)) (_)) (_(_()) 
| |  |_ _||_   _| 
| |__ | |   | |   
|____|___|  |_|   

I0822 15:19:40.357189 140224925210432 dev_server.py:80] Starting LIT server...
I0822 15:19:40.357330 140224925210432 caching.py:137] CachingModelWrapper 'sst': loading from /tmp/lit_data/sst.cache.pkl
I0822 15:19:40.424300 140224925210432 caching.py:96] Loaded cache (884 entries) from /tmp/lit_data/sst.cache.pkl
I0822 15:19:40.424804 140224925210432 wsgi_serving.py:39] 

Starting Server on port 5634
You can navigate to 0.0.0.0:5634

Error Log ( Traceback ):

I0822 15:20:40.686877 140224925210432 app.py:69] Request received: /get_info?
I0822 15:20:40.687858 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:20:40] "POST /get_info HTTP/1.1" 200 -
I0822 15:20:41.268086 140224925210432 app.py:69] Request received: /get_dataset?dataset_name=sst_dev
I0822 15:20:41.277388 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:20:41] "POST /get_dataset?dataset_name=sst_dev HTTP/1.1" 200 -
I0822 15:20:43.192993 140224925210432 app.py:69] Request received: /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=umap
I0822 15:20:43.701220 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:20:43.701397 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 872 inputs
I0822 15:20:43.701481 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:20:43.701582 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:20:43.701687 140224925210432 projection.py:207] Projection request: instance key: frozenset({('proj_kw', frozenset({('n_components', 3)})), ('model_name', 'sst'), ('dataset_name', 'sst_dev'), ('field_name', 'cls_emb')})
I0822 15:20:43.709572 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:20:43.709701 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 872 inputs
I0822 15:20:43.709782 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:20:43.709867 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:20:43.709962 140224925210432 projection.py:184] Creating new projection instance on 872 points
I0822 15:20:43.710053 140224925210432 caching.py:129] CachingModelWrapper 'frozenset({('proj_kw', frozenset({('n_components', 3)})), ('model_name', 'sst'), ('dataset_name', 'sst_dev'), ('field_name', 'cls_emb')})': no cache path specified, not loading.
I0822 15:20:43.712835 140224925210432 umap.py:38] UMAP input x_train: (872, 128)
I0822 15:20:53.559070 140224925210432 caching.py:211] CachingModelWrapper 'frozenset({('proj_kw', frozenset({('n_components', 3)})), ('model_name', 'sst'), ('dataset_name', 'sst_dev'), ('field_name', 'cls_emb')})': misses (dataset=): []
I0822 15:20:53.559272 140224925210432 caching.py:213] CachingModelWrapper 'frozenset({('proj_kw', frozenset({('n_components', 3)})), ('model_name', 'sst'), ('dataset_name', 'sst_dev'), ('field_name', 'cls_emb')})': 0 misses out of 872 inputs
I0822 15:20:53.559361 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:20:53.559463 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:20:53.563438 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:20:53] "POST /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=umap HTTP/1.1" 200 -
I0822 15:20:53.564406 140224925210432 app.py:69] Request received: /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds
I0822 15:20:53.568569 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:20:53.568676 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 872 inputs
I0822 15:20:53.568761 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:20:53.568853 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:20:53.568954 140224925210432 app.py:183] Requested types: ['MulticlassPreds']
I0822 15:20:53.569080 140224925210432 app.py:193] Will return keys: {'probas'}
I0822 15:20:53.573478 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:20:53] "POST /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds HTTP/1.1" 200 -
I0822 15:20:53.574222 140224925210432 app.py:69] Request received: /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=metrics
I0822 15:20:53.578125 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:20:53.578230 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 872 inputs
I0822 15:20:53.578310 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:20:53.578406 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:20:53.588980 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:20:53] "POST /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=metrics HTTP/1.1" 200 -
I0822 15:20:53.589661 140224925210432 app.py:69] Request received: /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds
I0822 15:20:53.593147 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:20:53.593254 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 872 inputs
I0822 15:20:53.593334 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:20:53.593420 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:20:53.593515 140224925210432 app.py:183] Requested types: ['MulticlassPreds']
I0822 15:20:53.593613 140224925210432 app.py:193] Will return keys: {'probas'}
I0822 15:20:53.597741 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:20:53] "POST /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds HTTP/1.1" 200 -
I0822 15:20:53.598444 140224925210432 app.py:69] Request received: /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=metrics
I0822 15:20:53.601932 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:20:53.602035 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 872 inputs
I0822 15:20:53.602116 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:20:53.602200 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:20:53.612254 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:20:53] "POST /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=metrics HTTP/1.1" 200 -
I0822 15:20:53.612898 140224925210432 app.py:69] Request received: /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds
I0822 15:20:53.616374 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:20:53.616477 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 872 inputs
I0822 15:20:53.616556 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:20:53.616641 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:20:53.616734 140224925210432 app.py:183] Requested types: ['MulticlassPreds']
I0822 15:20:53.616831 140224925210432 app.py:193] Will return keys: {'probas'}
I0822 15:20:53.621100 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:20:53] "POST /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds HTTP/1.1" 200 -
I0822 15:20:55.062833 140224925210432 app.py:69] Request received: /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds
I0822 15:20:55.566349 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:20:55.566529 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 872 inputs
I0822 15:20:55.566611 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:20:55.566708 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:20:55.566817 140224925210432 app.py:183] Requested types: ['MulticlassPreds']
I0822 15:20:55.566926 140224925210432 app.py:193] Will return keys: {'probas'}
I0822 15:20:55.571765 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:20:55] "POST /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds HTTP/1.1" 200 -
I0822 15:20:55.572703 140224925210432 app.py:69] Request received: /get_preds?model=sst&dataset_name=sst_dev&requested_types=RegressionScore
I0822 15:20:55.822110 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:20:55.822272 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 872 inputs
I0822 15:20:55.822356 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:20:55.822452 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:20:55.822559 140224925210432 app.py:183] Requested types: ['RegressionScore']
I0822 15:20:55.822671 140224925210432 app.py:193] Will return keys: set()
I0822 15:20:55.824508 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:20:55] "POST /get_preds?model=sst&dataset_name=sst_dev&requested_types=RegressionScore HTTP/1.1" 200 -
I0822 15:20:56.057429 140224925210432 app.py:69] Request received: /get_preds?model=sst&dataset_name=sst_dev&requested_types=Scalar
I0822 15:20:56.313553 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:20:56.313664 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 872 inputs
I0822 15:20:56.313746 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:20:56.313837 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:20:56.313934 140224925210432 app.py:183] Requested types: ['Scalar']
I0822 15:20:56.314047 140224925210432 app.py:193] Will return keys: set()
I0822 15:20:56.315773 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:20:56] "POST /get_preds?model=sst&dataset_name=sst_dev&requested_types=Scalar HTTP/1.1" 200 -
I0822 15:21:08.753991 140224925210432 app.py:69] Request received: /get_datapoint_ids?
I0822 15:21:08.754570 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:08] "POST /get_datapoint_ids HTTP/1.1" 200 -
I0822 15:21:10.000340 140224925210432 app.py:69] Request received: /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=metrics
I0822 15:21:10.511329 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): ['d70b4885afc1433da87688a15c32cf68']
I0822 15:21:10.511507 140224925210432 caching.py:213] CachingModelWrapper 'sst': 1 misses out of 873 inputs
I0822 15:21:10.511590 140224925210432 caching.py:218] Prepared 1 inputs for model
I0822 15:21:10.552725 140224925210432 caching.py:220] Received 1 predictions from model
I0822 15:21:10.563237 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:10] "POST /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=metrics HTTP/1.1" 200 -
I0822 15:21:10.748265 140224925210432 app.py:69] Request received: /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds
I0822 15:21:11.011022 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:21:11.011197 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 873 inputs
I0822 15:21:11.011280 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:21:11.011377 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:21:11.011486 140224925210432 app.py:183] Requested types: ['MulticlassPreds']
I0822 15:21:11.011596 140224925210432 app.py:193] Will return keys: {'probas'}
I0822 15:21:11.016319 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:11] "POST /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds HTTP/1.1" 200 -
I0822 15:21:11.017176 140224925210432 app.py:69] Request received: /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds
I0822 15:21:11.496760 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:21:11.496941 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 873 inputs
I0822 15:21:11.497025 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:21:11.497122 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:21:11.497264 140224925210432 app.py:183] Requested types: ['MulticlassPreds']
I0822 15:21:11.497374 140224925210432 app.py:193] Will return keys: {'probas'}
I0822 15:21:11.502411 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:11] "POST /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds HTTP/1.1" 200 -
I0822 15:21:11.503245 140224925210432 app.py:69] Request received: /get_preds?model=sst&dataset_name=sst_dev&requested_types=RegressionScore
I0822 15:21:11.760120 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:21:11.760298 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 873 inputs
I0822 15:21:11.760380 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:21:11.760478 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:21:11.760585 140224925210432 app.py:183] Requested types: ['RegressionScore']
I0822 15:21:11.760695 140224925210432 app.py:193] Will return keys: set()
I0822 15:21:11.762583 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:11] "POST /get_preds?model=sst&dataset_name=sst_dev&requested_types=RegressionScore HTTP/1.1" 200 -
I0822 15:21:11.772038 140224925210432 app.py:69] Request received: /get_preds?model=sst&dataset_name=sst_dev&requested_types=Scalar
I0822 15:21:12.243162 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:21:12.243347 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 873 inputs
I0822 15:21:12.243430 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:21:12.243527 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:21:12.243634 140224925210432 app.py:183] Requested types: ['Scalar']
I0822 15:21:12.243743 140224925210432 app.py:193] Will return keys: set()
I0822 15:21:12.245831 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:12] "POST /get_preds?model=sst&dataset_name=sst_dev&requested_types=Scalar HTTP/1.1" 200 -
I0822 15:21:12.246693 140224925210432 app.py:69] Request received: /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=metrics
I0822 15:21:12.246981 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:21:12.247077 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 1 inputs
I0822 15:21:12.247156 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:21:12.247241 140224925210432 caching.py:220] Received 0 predictions from model
/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/numpy/lib/function_base.py:393: RuntimeWarning: Mean of empty slice.
  avg = a.mean(axis)
/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/numpy/core/_methods.py:161: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1221: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1221: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 due to no true samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1465: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  average, "true nor predicted", 'F-score is', len(true_sum)
I0822 15:21:12.252254 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:12] "POST /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=metrics HTTP/1.1" 200 -
I0822 15:21:12.498321 140224925210432 app.py:69] Request received: /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds
I0822 15:21:12.498716 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:21:12.498813 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 1 inputs
I0822 15:21:12.498892 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:21:12.498981 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:21:12.499068 140224925210432 app.py:183] Requested types: ['MulticlassPreds']
I0822 15:21:12.499170 140224925210432 app.py:193] Will return keys: {'probas'}
I0822 15:21:12.499482 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:12] "POST /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds HTTP/1.1" 200 -
I0822 15:21:12.996581 140224925210432 app.py:69] Request received: /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=counterfactual%20explainer
I0822 15:21:12.997049 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:21:12.997206 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 1 inputs
I0822 15:21:12.997319 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:21:12.997461 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:21:12.997652 140224925210432 lemon_explainer.py:87] Found text fields for LEMON attribution: ['sentence']
I0822 15:21:12.997792 140224925210432 lemon_explainer.py:108] Explaining: This won't work
The exact solution is  x = 0                              
I0822 15:21:13.000373 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:13] "POST /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=counterfactual%20explainer HTTP/1.1" 200 -
I0822 15:21:13.538210 140224925210432 app.py:69] Request received: /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds
I0822 15:21:13.538661 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:21:13.538776 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 1 inputs
I0822 15:21:13.538902 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:21:13.539038 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:21:13.539162 140224925210432 app.py:183] Requested types: ['MulticlassPreds']
I0822 15:21:13.539313 140224925210432 app.py:193] Will return keys: {'probas'}
I0822 15:21:13.539734 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:13] "POST /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds HTTP/1.1" 200 -
I0822 15:21:13.787961 140224925210432 app.py:69] Request received: /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds
I0822 15:21:13.788417 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:21:13.788529 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 1 inputs
I0822 15:21:13.788653 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:21:13.788795 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:21:13.788922 140224925210432 app.py:183] Requested types: ['MulticlassPreds']
I0822 15:21:13.789069 140224925210432 app.py:193] Will return keys: {'probas'}
I0822 15:21:13.789474 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:13] "POST /get_preds?model=sst&dataset_name=sst_dev&requested_types=MulticlassPreds HTTP/1.1" 200 -
I0822 15:21:14.284508 140224925210432 app.py:69] Request received: /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=grad_norm
I0822 15:21:14.284989 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:21:14.285106 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 1 inputs
I0822 15:21:14.285258 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:21:14.285397 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:21:14.285616 140224925210432 gradient_maps.py:65] Found fields for gradient attribution: ['token_grad_sentence']
/opt/lit/lit_nlp/components/gradient_maps.py:51: RuntimeWarning: invalid value encountered in true_divide
  grad_norm /= np.sum(grad_norm)
I0822 15:21:14.286377 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:14] "POST /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=grad_norm HTTP/1.1" 200 -
I0822 15:21:14.824009 140224925210432 app.py:69] Request received: /get_preds?model=sst&dataset_name=sst_dev&requested_types=Tokens,AttentionHeads
I0822 15:21:14.824451 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:21:14.824565 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 1 inputs
I0822 15:21:14.824690 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:21:14.824824 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:21:14.824951 140224925210432 app.py:183] Requested types: ['Tokens', 'AttentionHeads']
I0822 15:21:14.825211 140224925210432 app.py:193] Will return keys: {'layer_1/attention', 'tokens_sentence', 'tokens', 'layer_0/attention'}
I0822 15:21:14.825782 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:14] "POST /get_preds?model=sst&dataset_name=sst_dev&requested_types=Tokens,AttentionHeads HTTP/1.1" 200 -
I0822 15:21:15.072378 140224925210432 app.py:69] Request received: /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=counterfactual%20explainer
I0822 15:21:15.072764 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:21:15.072861 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 1 inputs
I0822 15:21:15.072937 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:21:15.073023 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:21:15.073159 140224925210432 lemon_explainer.py:87] Found text fields for LEMON attribution: ['sentence']
I0822 15:21:15.073270 140224925210432 lemon_explainer.py:108] Explaining: This won't work
The exact solution is  x = 0                              
I0822 15:21:15.075521 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:15] "POST /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=counterfactual%20explainer HTTP/1.1" 200 -
I0822 15:21:15.817224 140224925210432 app.py:69] Request received: /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=umap
I0822 15:21:16.081932 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:21:16.082101 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 873 inputs
I0822 15:21:16.082181 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:21:16.082275 140224925210432 caching.py:220] Received 0 predictions from model
I0822 15:21:16.082371 140224925210432 projection.py:207] Projection request: instance key: frozenset({('proj_kw', frozenset({('n_components', 3)})), ('model_name', 'sst'), ('dataset_name', 'sst_dev'), ('field_name', 'cls_emb')})
I0822 15:21:16.084426 140224925210432 caching.py:211] CachingModelWrapper 'frozenset({('proj_kw', frozenset({('n_components', 3)})), ('model_name', 'sst'), ('dataset_name', 'sst_dev'), ('field_name', 'cls_emb')})': misses (dataset=): ['d70b4885afc1433da87688a15c32cf68']
I0822 15:21:16.084517 140224925210432 caching.py:213] CachingModelWrapper 'frozenset({('proj_kw', frozenset({('n_components', 3)})), ('model_name', 'sst'), ('dataset_name', 'sst_dev'), ('field_name', 'cls_emb')})': 1 misses out of 873 inputs
I0822 15:21:16.084591 140224925210432 caching.py:218] Prepared 1 inputs for model
E0822 15:21:16.088476 140224925210432 wsgi_app.py:210] Uncaught error: Incompatible dimension for X and Y matrices: X.shape[1] == 2 while Y.shape[1] == 128 

 Traceback (most recent call last):
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/umap/umap_.py", line 2056, in transform
    X, self._raw_data, metric=_m, **self._metric_kwds
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/utils/validation.py", line 73, in inner_f
    return f(**kwargs)
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/metrics/pairwise.py", line 1775, in pairwise_distances
    return _parallel_pairwise(X, Y, func, n_jobs, **kwds)
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/metrics/pairwise.py", line 1359, in _parallel_pairwise
    return func(X, Y, **kwds)
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/metrics/pairwise.py", line 1379, in _pairwise_callable
    X, Y = check_pairwise_arrays(X, Y, force_all_finite=force_all_finite)
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/utils/validation.py", line 73, in inner_f
    return f(**kwargs)
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/metrics/pairwise.py", line 160, in check_pairwise_arrays
    X.shape[1], Y.shape[1]))
ValueError: Incompatible dimension for X and Y matrices: X.shape[1] == 2 while Y.shape[1] == 128

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/lit/lit_nlp/lib/wsgi_app.py", line 193, in __call__
    return self._ServeCustomHandler(request, clean_path)(environ,
  File "/opt/lit/lit_nlp/lib/wsgi_app.py", line 178, in _ServeCustomHandler
    return self._handlers[clean_path](self, request)
  File "/opt/lit/lit_nlp/app.py", line 77, in _handler
    outputs = fn(data, **kw)
  File "/opt/lit/lit_nlp/app.py", line 240, in _get_interpretations
    config=data.get('config'))
  File "/opt/lit/lit_nlp/components/projection.py", line 198, in run_with_metadata
    return self._run_with_metadata(*args, **kw)
  File "/opt/lit/lit_nlp/components/projection.py", line 216, in _run_with_metadata
    model_outputs)
  File "/opt/lit/lit_nlp/components/projection.py", line 131, in run_with_metadata
    return self._run(model, indexed_inputs, model_outputs, do_fit=False)
  File "/opt/lit/lit_nlp/components/projection.py", line 121, in _run
    converted_inputs, dataset_name="")
  File "/opt/lit/lit_nlp/lib/caching.py", line 219, in predict_with_metadata
    model_preds = list(self._model.predict_with_metadata(model_inputs))
  File "/opt/lit/lit_nlp/api/model.py", line 177, in <genexpr>
    results = (scrub_numpy_refs(res) for res in results)
  File "/opt/lit/lit_nlp/api/model.py", line 192, in _batched_predict
    yield from self.predict_minibatch(minibatch, **kw)
  File "/opt/lit/lit_nlp/components/umap.py", line 49, in predict_minibatch
    zs = self._umap.transform(x)
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/umap/umap_.py", line 2063, in transform
    kwds=self._metric_kwds,
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/umap/distances.py", line 1260, in pairwise_special_metric
    return pairwise_distances(X, Y, metric=_partial_metric)
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/utils/validation.py", line 73, in inner_f
    return f(**kwargs)
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/metrics/pairwise.py", line 1775, in pairwise_distances
    return _parallel_pairwise(X, Y, func, n_jobs, **kwds)
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/metrics/pairwise.py", line 1359, in _parallel_pairwise
    return func(X, Y, **kwds)
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/metrics/pairwise.py", line 1379, in _pairwise_callable
    X, Y = check_pairwise_arrays(X, Y, force_all_finite=force_all_finite)
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/utils/validation.py", line 73, in inner_f
    return f(**kwargs)
  File "/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/metrics/pairwise.py", line 160, in check_pairwise_arrays
    X.shape[1], Y.shape[1]))
ValueError: Incompatible dimension for X and Y matrices: X.shape[1] == 2 while Y.shape[1] == 128

I0822 15:21:16.089119 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:16] "POST /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=umap HTTP/1.1" 500 -
I0822 15:21:16.089920 140224925210432 app.py:69] Request received: /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=metrics
I0822 15:21:16.090191 140224925210432 caching.py:211] CachingModelWrapper 'sst': misses (dataset=sst_dev): []
I0822 15:21:16.090284 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 1 inputs
I0822 15:21:16.090358 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:21:16.090440 140224925210432 caching.py:220] Received 0 predictions from model
/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/numpy/lib/function_base.py:393: RuntimeWarning: Mean of empty slice.
  avg = a.mean(axis)
/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/numpy/core/_methods.py:161: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1221: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1221: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 due to no true samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/opt/conda/envs/lit-nlp/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1465: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  average, "true nor predicted", 'F-score is', len(true_sum)
I0822 15:21:16.093940 140224925210432 _internal.py:113] 172.17.0.1 - - [22/Aug/2020 15:21:16] "POST /get_interpretations?model=sst&dataset_name=sst_dev&interpreter=metrics HTTP/1.1" 200 -

Kindly help.

Thanks

tolga-b commented 4 years ago

Hi rwikdutta, It seems like all the requests in your case have been loaded from the cache of a previous run:

I0822 15:19:40.357330 140224925210432 caching.py:137] CachingModelWrapper 'sst': loading from /tmp/lit_data/sst.cache.pkl
I0822 15:19:40.424300 140224925210432 caching.py:96] Loaded cache (884 entries) from /tmp/lit_data/sst.cache.pkl

and 

I0822 15:20:43.701397 140224925210432 caching.py:213] CachingModelWrapper 'sst': 0 misses out of 872 inputs
I0822 15:20:43.701481 140224925210432 caching.py:218] Prepared 0 inputs for model
I0822 15:20:43.701582 140224925210432 caching.py:220] Received 0 predictions from model

It could be that the cache contains outputs from the previous model and the first time you call the new model is when you add a new datapoint. I think what is happening is that the cached UMAP embedding dimension (128) does not match the new model's embedding dimension (2?). This may happen if you gave the new model the same name as the previous model here: https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/quickstart_sst_demo.py#L51 ("sst" for both) . Could you please clear "/tmp/lit_data" directory and run again only with the new model?

rwikdutta commented 4 years ago

Thanks for the clarification.