maxjcohen
commented
2 years ago Hi, this seems to be an issue with Pandas, as you can see in the last stack of the Traceback. Did you try to feed a pandas Dataframe directly to the trainer ? You most likely need to modify the dataloader class to match your dataset.
I currently use a custom dataset defined here. For an example of dataloader definition, see https://github.com/maxjcohen/transformer/blob/bea1c92a336583d28eb5869be312055747b67c73/training.py#L52-L57
The documentation for both dataset and dataloader can be found, along with examples, on the PyTorch documentation.
shathaa1983
Hi, I am trying to use a univariate time series dataset. I got this error:
KeyError Traceback (most recent call last) ~\anaconda3\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance) 3079 try: -> 3080 return self._engine.get_loc(casted_key) 3081 except KeyError as err:
pandas_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
pandas_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
KeyError: 521170
The above exception was the direct cause of the following exception:
KeyError Traceback (most recent call last) ~\AppData\Local\Temp/ipykernel_17036/184118730.py in 9 running_loss = 0 10 with tqdm(total=len(dataloader_train.dataset), desc=f"[Epoch {idx_epoch+1:3d}/{EPOCHS}]") as pbar: ---> 11 for idx_batch, (x, y) in enumerate(dataloader_train): 12 try: 13 print(f'idx_batch {dataloader_train} is {enumerate[dataloader_train]}')
~\anaconda3\lib\site-packages\torch\utils\data\dataloader.py in next(self) 519 if self._sampler_iter is None: 520 self._reset() --> 521 data = self._next_data() 522 self._num_yielded += 1 523 if self._dataset_kind == _DatasetKind.Iterable and \
~\anaconda3\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self) 559 def _next_data(self): 560 index = self._next_index() # may raise StopIteration --> 561 data = self._dataset_fetcher.fetch(index) # may raise StopIteration 562 if self._pin_memory: 563 data = _utils.pin_memory.pin_memory(data)
~\anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py in fetch(self, possibly_batched_index) 42 def fetch(self, possibly_batched_index): 43 if self.auto_collation: ---> 44 data = [self.dataset[idx] for idx in possibly_batched_index] 45 else: 46 data = self.dataset[possibly_batched_index]
~\anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py in (.0) 42 def fetch(self, possibly_batched_index): 43 if self.auto_collation: ---> 44 data = [self.dataset[idx] for idx in possibly_batched_index] 45 else: 46 data = self.dataset[possibly_batched_index]
~\anaconda3\lib\site-packages\torch\utils\data\dataset.py in getitem(self, idx) 309 310 def getitem(self, idx): --> 311 return self.dataset[self.indices[idx]] 312 313 def len(self):
~\anaconda3\lib\site-packages\pandas\core\frame.py in getitem(self, key) 3022 if self.columns.nlevels > 1: 3023 return self._getitem_multilevel(key) -> 3024 indexer = self.columns.get_loc(key) 3025 if is_integer(indexer): 3026 indexer = [indexer]
~\anaconda3\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance) 3080 return self._engine.get_loc(casted_key) 3081 except KeyError as err: -> 3082 raise KeyError(key) from err 3083 3084 if tolerance is not None:
KeyError: 521170
I'd appreciate it if you let me know if your code is suitable for the univariate time series. And how to solve this error? I used the code in this link: https://github.com/maxjcohen/transformer
Thanks