Closed averrows closed 2 months ago
Hi @averrows , please ignore the dataloader part for Thoughtviz dataset. In the training code we have imported the pickle file directly without using the dataloader.
It is still being called when I try to run cmd.txt's script to train the EEGStyleGAN-ADA. Or should I run something else?
@averrows , I hope you are running EEGStyleGAN-ADA Thoughtviz code not the cvpr40 dataset one. There are two different version of the code.
This is the error I got. The dataset.py is being called here.
File "/raid/data/m13520100/TA/EEGStyleGAN-ADA/EEGStyleGAN-ADA_ThoughtViz/dnnlib/util.py", line 289, in construct_class_by_name
return call_func_by_name(*args, func_name=class_name, **kwargs)
File "/raid/data/m13520100/TA/EEGStyleGAN-ADA/EEGStyleGAN-ADA_ThoughtViz/dnnlib/util.py", line 284, in call_func_by_name
return func_obj(*args, **kwargs)
File "/raid/data/m13520100/TA/EEGStyleGAN-ADA/EEGStyleGAN-ADA_ThoughtViz/training/dataset.py", line 277, in __init__
loaded_array = np.load(path, allow_pickle=True)
File "/raid/data/m13520100/env/lib/python3.8/site-packages/numpy/lib/npyio.py", line 443, in load
raise pickle.UnpicklingError(
_pickle.UnpicklingError: Failed to interpret file '../../eeg/char/data.pkl' as a pickle
@averrows Based on the error, there might be some problem with either dataset file or numpy version. Can you load that pickle file and test if there is no fault with pickle file and numpy version.
The numpy version I am using is 1.21.0, the same version the anaconda yml file specify. I got the dataset from this https://iitgnacin-my.sharepoint.com/personal/19210048_iitgn_ac_in/_layouts/15/onedrive.aspx?id=%2Fpersonal%2F19210048%5Fiitgn%5Fac%5Fin%2FDocuments%2FGdrive%2FPhD%2FCodes%2FDatasets%2Fthoughtviz%5Feeg%5Fimage%5Fchar%5Fdigit%2Ezip&parent=%2Fpersonal%2F19210048%5Fiitgn%5Fac%5Fin%2FDocuments%2FGdrive%2FPhD%2FCodes%2FDatasets&ga=1 that you share in other issue. So it should be correct.
However, when I load it with latin1
encoding. The data is loaded and the keys in of the data is dict_keys(['x_test', 'x_train', 'y_train', 'y_test'])
import numpy as np
data = np.load("../../eeg/char/data.pkl", allow_pickle=True, encoding="latin1")
print(data.keys())
print(data.shape)
I initially wanted to map the keys with the index the program try to access. However, I can't do that because it has different structure (That's why I am asking the initial question).
I am trying to train the model using Thoughtviz dataset. The Thoughtviz's pickle structure is a dictionary with 4 keys: x_test, x_train, y_train, and x_test. While the EEG2ImageDataset class index the pickle file with index 0 as EEG, index 1 as Labels, and index 2 as Images. This difference makes me confused. Where should I retrieve the dataset file?
Are there steps that I missed or is it a device-specific issue?
@averrows can you add
encoding="latin1"at line 277 of dataset.py while loading the pickle and check if it works.
File "train.py", line 489, in main
run_desc, args = setup_training_loop_kwargs(**config_kwargs)
File "train.py", line 113, in setup_training_loop_kwargs
training_set = dnnlib.util.construct_class_by_name(**args.training_set_kwargs) # subclass of training.dataset.Dataset
File "/raid/data/m13520100/TA/EEGStyleGAN-ADA/EEGStyleGAN-ADA_ThoughtViz/dnnlib/util.py", line 289, in construct_class_by_name
return call_func_by_name(*args, func_name=class_name, **kwargs)
File "/raid/data/m13520100/TA/EEGStyleGAN-ADA/EEGStyleGAN-ADA_ThoughtViz/dnnlib/util.py", line 284, in call_func_by_name
return func_obj(*args, **kwargs)
File "/raid/data/m13520100/TA/EEGStyleGAN-ADA/EEGStyleGAN-ADA_ThoughtViz/training/dataset.py", line 279, in __init__
eeg = np.float32(np.squeeze(loaded_array[0], axis=-1).T)
KeyError: 0
Here is the issue. The content of the pickle is a dictionary with dict_keys(['x_test', 'x_train', 'y_train', 'y_test'])
as its keys. So when the program try to access [0] it returns KeyError.
Did I download the wrong pickle file?
@prajwalsingh is there something I can do to solve this?
Hi @averrows , I checked the EEGStyleGAN training code and found that, indeed, for training it, we created a numpy file with eeg, label, and image information from the pickle file of the thoughtviz dataset. You can find the numpy format of ThoughtViz dataset here: [link]
Thank you for your assistance! it is working perfectly
I am trying to train the model using Thoughtviz dataset. The Thoughtviz's pickle structure is a dictionary with 4 keys: x_test, x_train, y_train, and x_test. While the EEG2ImageDataset class index the pickle file with index 0 as EEG, index 1 as Labels, and index 2 as Images. This difference makes me confused. Where should I retrieve the dataset file?