KeyError in Transfer_Learning_Demo_Teachers_Forcing.ipynb

[stephanflemming]

Hi,

Transfer_Learning_Demo_Teachers_Forcing.ipynb produces the following error. I tried branch 3.0 and 3.1.

Thanks in advance, Stephan

[...]
100%|#####################################| 4064/4064 [1:50:52<00:00,  1.25s/it]
100%|#####################################| 4064/4064 [1:50:52<00:00,  1.64s/it]
Traceback (most recent call last):
  File "/[...]/Reinvent-3.1/input.py", line 21, in <module>
    manager.run()
  File "/[...]/Reinvent-3.1/running_modes/manager.py", line 17, in run
    runner.run()
  File "/[...]/Reinvent-3.1/running_modes/transfer_learning/transfer_learning_runner.py", line 40, in run
    self._train_epoch(epoch, self._config.input_smiles_path)
  File "/[...]/Reinvent-3.1/running_modes/transfer_learning/transfer_learning_runner.py", line 61, in _train_epoch
    self._calculate_stats_and_update_learning_rate(epoch, model_path)
  File "/[...]/Reinvent-3.1/running_modes/transfer_learning/transfer_learning_runner.py", line 94, in _calculate_stats_and_update_learning_rate
    validation_set_path=self._config.validation_smiles_path)
  File "/[...]/Reinvent-3.1/running_modes/transfer_learning/adaptive_learning_rate.py", line 79, in collect_stats
    training_nlls = self._calc_nlls(model, training_set_path, self._config.sample_size)
  File "/[...]/Reinvent-3.1/running_modes/transfer_learning/adaptive_learning_rate.py", line 105, in _calc_nlls
    list(md.calculate_nlls_from_model(model, smiles)[0]))
  File "/anaconda/envs/reinvent.v3.0/lib/python3.7/site-packages/reinvent_models/reinvent_core/models/dataset.py", line 49, in _iterator
    for batch in dataloader:
  File "/anaconda/envs/reinvent.v3.0/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/anaconda/envs/reinvent.v3.0/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/anaconda/envs/reinvent.v3.0/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/anaconda/envs/reinvent.v3.0/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/anaconda/envs/reinvent.v3.0/lib/python3.7/site-packages/reinvent_models/reinvent_core/models/dataset.py", line 22, in __getitem__
    encoded = self._vocabulary.encode(tokens)
  File "/anaconda/envs/reinvent.v3.0/lib/python3.7/site-packages/reinvent_models/reinvent_core/models/vocabulary.py", line 59, in encode
    vocab_index[i] = self._tokens[token]
KeyError: '\x00'

JSON

{
    "logging": {
        "job_id": "demo",
        "job_name": "Transfer Learning demo",
        "logging_path": "[...]/transferlearning_log",
        "recipient": "local",
        "sender": "http://127.0.0.1"
    },
    "parameters": {
        "adaptive_lr_config": {
            "average_steps": 4,
            "gamma": 0.8,
            "min": 1e-05,
            "mode": "adaptive",
            "patience": 8,
            "restart_times": 0,
            "restart_value": 1e-05,
            "sample_size": 10000,
            "start": 0.0005,
            "step": 1,
            "threshold": 0.0001
        },
        "batch_size": 128,
        "input_model_path": "[...]/data/random.prior.new",
        "input_smiles_path": "[...]/data/chembl.filtered.smi",
        "num_epochs": 3,
        "output_model_path": "[...]/results/create_prior/chembl.prior",
        "randomize": true,
        "save_every_n_epochs": 1,
        "standardize": false
    },
    "run_type": "transfer_learning",
    "version": 3
}

[patronov]

Hi, it seems that there is something unusual with the input. It finds the following binary key in the file '\x00' If I understand correctly it does two epochs before failing? Is it always after the second epoch?

[stephanflemming]

The mentioned error appears after nearly 2h calculation time. The files I used (from the ReinventCommunity repository) can be found here.

[simmed00]

Could this be a problem with the dataset chembl_filtered.smi? I have exactly the same issue. However, after I change the dataset to be chembl_mini.smi, it runs without error

[GuoJeff]

Hi @stephanflemming and @simmed00,

Sorry for the delay. The error is due to a problem with the provided dataset as @simmed00 pointed out above. This has now been fixed in the latest code release. If you pull the Reinvent and ReinventCommunity updates, this issue should be resolved.