search

private-synthetic-data-social-sciences / explore_dpcgans

Apache License 2.0
0 stars 0 forks source link

Cannot sample on the gpu from pickled model #1

Open f-hafner opened 1 year ago

f-hafner commented 1 year ago

On snellius, I cannot sample from the pickled models. I suppose the problem is some difference in dependencies in my environment, and the environment where the model was trained.

I'm putting this here for documentation, but I think it's not relevant anymore (because we need to train new models anyway.

In snellius login node:

module load 2021
module load Python/3.9.5-GCCcore-10.3.0-bare # the non-bare version comes with an older poetry version
curl -sSL https://install.python-poetry.org | python3 - --git https://github.com/python-poetry/poetry.git@master # as of 17/10/23. https://github.com/python-poetry/poetry/issues/1917#issuecomment-1745782011

cd projects/gans/explore_dpcgans
poetry config virtualenvs.in-project true
poetry install --no-root

Then run an interactive job:

srun -p gpu --nodes=1 --gpus=1 --time=15:00 --pty /bin/bash

cd projects/gans/explore_dpcgans
module load 2021
module load Python/3.9.5-GCCcore-10.3.0-bare

poetry shell 

python sample_dpcgans.py # raises errors

The file sample_dpcgans.py is as follows:

import torch 
assert torch.cuda.is_available(), "cuda not available"
import pickle 

filename = "/home/fhafner/gans/datasets/Adult/DPCGANS/dpcgans_train_2000.pkl"
filename = "/gpfs/home2/hafner/projects/gans/datasets/Adult/DPCGANS/dpcgans_train_2000.pkl"
# filename = "/gpfs/home2/hafner/projects/gans/datasets/Census/DPCGANS/dpcgans_train_2000.pkl"

# load the model
with open(filename, "rb") as f:
    model = pickle.load(f)

# try to sample
model.sample()

The error I get:

Traceback (most recent call last):
  File "/gpfs/home2/hafner/projects/gans/explore_dpcgans/sample_dpcgans.py", line 15, in <module>
    model.sample()
  File "/gpfs/home2/hafner/projects/gans/explore_dpcgans/.venv/lib/python3.9/site-packages/dp_cgans/base.py", line 442, in sample
    return self._sample_batch(num_rows, max_retries, max_rows_multiplier)
  File "/gpfs/home2/hafner/projects/gans/explore_dpcgans/.venv/lib/python3.9/site-packages/dp_cgans/base.py", line 299, in _sample_batch
    sampled, num_valid = self._sample_rows(
  File "/gpfs/home2/hafner/projects/gans/explore_dpcgans/.venv/lib/python3.9/site-packages/dp_cgans/base.py", line 228, in _sample_rows
    sampled = self._sample(num_rows)
  File "/gpfs/home2/hafner/projects/gans/explore_dpcgans/.venv/lib/python3.9/site-packages/dp_cgans/dp_cgan_init.py", line 80, in _sample
    return self._model.sample(num_rows)
  File "/gpfs/home2/hafner/projects/gans/explore_dpcgans/.venv/lib/python3.9/site-packages/dp_cgans/synthesizers/dp_cgan.py", line 664, in sample
    return self._transformer.inverse_transform(data)
  File "/gpfs/home2/hafner/projects/gans/explore_dpcgans/.venv/lib/python3.9/site-packages/dp_cgans/data_transformer.py", line 198, in inverse_transform
    recovered_column_data = self._inverse_transform_discrete(
  File "/gpfs/home2/hafner/projects/gans/explore_dpcgans/.venv/lib/python3.9/site-packages/dp_cgans/data_transformer.py", line 178, in _inverse_transform_discrete
    return ohe.reverse_transform(data)[column_transform_info.column_name]
  File "/gpfs/home2/hafner/projects/gans/explore_dpcgans/.venv/lib/python3.9/site-packages/rdt/transformers/base.py", line 279, in reverse_transform
    if any(column not in data.columns for column in self.output_columns):
TypeError: 'NoneType' object is not iterable

Moreover, poetry shell gives the following output:

Spawning shell within /gpfs/home2/hafner/projects/gans/explore_dpcgans/.venv
. /gpfs/home2/hafner/projects/gans/explore_dpcgans/.venv/bin/activate
flatpak: /sw/arch/Centos8/EB_production/2021/software/XZ/5.2.5-GCCcore-10.3.0/lib/liblzma.so.5: version `XZ_5.2' not found (required by /lib64/libarchive.so.13)
[hafner@gcn48 explore_dpcgans]$ . /gpfs/home2/hafner/projects/gans/explore_dpcgans/.venv/bin/activate
f-hafner commented 1 year ago

Earlier on, I tried investigate this. What I found:

  1. in the rdt.transformers.categorical.OneHotEncodingTransformer, (though method is defined in .base) the output columns are not defined: 
    
(Pdb) self.get_output_columns()
['None.value0', 'None.value1', 'None.value2', 'None.value3', 'None.value4', 'None.value5', 'None.value6']
(Pdb) data.columns
Index(['None.value0', 'None.value1', 'None.value2', 'None.value3',
   'None.value4', 'None.value5', 'None.value6'],
  dtype='object')
(Pdb) data.head()
None.value0   None.value1   None.value2   None.value3   None.value4   None.value5   None.value6
0  8.544042e-01  2.655355e-17  7.015797e-10  1.271636e-01  1.522267e-02  3.209622e-03  4.481079e-38
1  5.825917e-25  8.926271e-43  1.741067e-35  3.992792e-23  2.501956e-29  5.874925e-33  1.000000e+00
2  1.000000e+00  2.567566e-31  9.168337e-29  3.365345e-13  1.293156e-11  3.809055e-27  0.000000e+00
3  1.000000e+00  3.928383e-17  1.207312e-22  4.372707e-18  5.076192e-15  3.916351e-18  3.940451e-42
4  1.000000e+00  1.901732e-15  3.121897e-34  8.550410e-18  1.568426e-16  2.234726e-16  4.063766e-43
(Pdb) self.output_columns is None
True


2. in dp_cgans.data_transformer._inverse_transform_discrete()
- `ohe.get_input_columns() is None ` is True 
- inherits from `self._column_transform_info_list` in line 189 of data_transformer.py