Getting error "Pickl Generator"

[sarangs-ntnu]

I am getting error as below when using model sweep for either "full" or "high_memory".

TypeError: cannot pickle 'generator' object

[manujosephv]

Can you share a reproducible example here? We need more information to understand what's the issue..

[sarangs-ntnu]

Hi.

I am following the sample code available on the website for model sweep.

For the lite and standard model list, it is working fine. But, for the rest of the two I got the error shared on github.

Best,

Sarang Shaikh

PhD Researcher Department of Information Security and Communication Technology Norwegian University of Science and Technology (NTNU) Teknologivegen 22, 2815 Gjøvik, Norway Phone: +4792041857 Email: @.***

---

From: Manu Joseph V @.> Sent: Monday, August 19, 2024 7:12 AM To: manujosephv/pytorch_tabular @.> Cc: Sarang Shaikh @.>; Author @.> Subject: Re: [manujosephv/pytorch_tabular] Getting error "Pickl Generator" (Issue #469)

Can you share a reproducible example here? We need more information to understand what's the issue..

— Reply to this email directly, view it on GitHubhttps://github.com/manujosephv/pytorch_tabular/issues/469#issuecomment-2295683195, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A37NKSLF6L5VGGZS52VSYFLZSF5CPAVCNFSM6AAAAABMWXGUISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJVGY4DGMJZGU. You are receiving this because you authored the thread.Message ID: @.***>

[jareciog]

Hi Manu. Thank you very much for your fantastic library. It's awesome.

I also get the same error for "full" or "high_memory" (others work fine). It is a simple classification dataset that works perfectly when training models individually, but model sweep fails.

I'm running an ipynb notebook (this could be the problem) over python version 3.9.6. This is log:

---

TypeError Traceback (most recent call last) Cell In[17], line 4 2 with warnings.catch_warnings(): 3 warnings.simplefilter("ignore") ----> 4 sweep_df, best_model = model_sweep( 5 task="classification", # One of "classification", "regression" 6 train=train, 7 test=test, 8 data_config=data_config, 9 optimizer_config=optimizer_config, 10 trainer_config=trainer_config, 11 model_list="full", 12 common_model_args=dict(head="LinearHead", head_config=head_config), 13 metrics=["accuracy", "f1_score"], 14 metrics_params=[{}, {"average": "macro"}], 15 metrics_prob_input=[False, True], 16 rank_metric=("accuracy", "higher_is_better"), 17 progress_bar=True, 18 verbose=False, 19 suppress_lightning_logger=True, 20 )

File ~/investigacion/inv-tests/lib/python3.9/site-packages/pytorch_tabular/tabular_model_sweep.py:279, in model_sweep(task, train, test, data_config, optimizer_config, trainer_config, model_list, metrics, metrics_params, metrics_prob_input, validation, experiment_config, common_model_args, rank_metric, return_best_model, seed, ignore_oom, progress_bar, verbose, suppress_lightning_logger) 277 common_model_args[arg] = locals()[arg] 278 if isinstance(model_list, str): --> 279 model_list = copy.deepcopy(MODEL_SWEEP_PRESETS[model_list]) 280 model_list = [ 281 ( 282 getattr(models, model_config[0])(task=task, model_config[1], common_model_args) (...) 290 for model_config in model_list 291 ] 293 def _init_tabular_model(m):

File /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py:161, in deepcopy(x, memo, _nil) 159 reductor = getattr(x, "__reduce_ex", None) 160 if reductor is not None: --> 161 rv = reductor(4) 162 else: 163 reductor = getattr(x, "reduce__", None)

TypeError: cannot pickle 'generator' object

Thank you!

[jareciog]

Hi again. I have tried as a py script and it also fails:

Traceback (most recent call last): File "investigacion/inv-tests/test-sweep.py", line 79, in sweep_df, best_model = model_sweep( File "investigacion/inv-tests/lib/python3.9/site-packages/pytorch_tabular/tabular_model_sweep.py", line 279, in model_sweep model_list = copy.deepcopy(MODEL_SWEEP_PRESETS[model_list]) File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 161, in deepcopy rv = reductor(4) TypeError: cannot pickle 'generator' object

[manujosephv]

@ProgramadorArtificial Do you want to take a look at this? Does look like one of the models in the Full and High Memory configurations have some generator which python can't pickle? I'll take a couple more months to have a bit more free time to delve into this.

[ProgramadorArtificial]

Hello! The pytorch_tabular library in PIP is not updated. Please clone the main repository and run "pip install .[extra]" to install the latest version of the library, this will stop the error.

PS: @manujosephv, when possible update the library in PIP, please :)