ValueError: optimizer got an empty parameter list

[CYH4157]

Describe the issue: I have a little trouble with my rewritting LSTM model in this "Retiarii_example_one-shot_NAS.ipynb" tutorial code, I have set up the optimizer, but it will not read in. what could be the reason? Thank you all for your help!!

Environment:

• NNI version: 2.7
• Training service (local|remote|pai|aml|etc): local
• Client OS: ubuntu20.04
• Server OS (for remote mode only): no
• Python version: 3.8
• PyTorch/TensorFlow version: pytorch==1.11
• Is conda/virtualenv/venv used?: no
• Is running in Docker?: docker

Configuration:

• Experiment config (remember to remove secrets!):
• Search space:

Log message:

• nnimanager.log:
• dispatcher.log:
• nnictl stdout and stderr:

import torch.nn.functional as F
import nni.retiarii.nn.pytorch as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.sofa_network_lstm1 = nn.LSTM(input_size=13, hidden_size=13, bidirectional=True, batch_first=True)

        neural_tuning = nn.ValueChoice([64, 128, 256], label='hidden_1')

        self.sofa_network_lstm2 = nn.LSTM(input_size=26, hidden_size=neural_tuning, bidirectional=True, batch_first=True)

        self.sofa_network_dropout = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75], label='hidden_2'))
        self.sofa_network_lstm3 = nn.LSTM(input_size=neural_tuning*2, hidden_size=13, bidirectional=True, batch_first=True)

        self.sofa_network_bn = nn.BatchNorm1d(1)
        self.sofa_network_dropout2 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75], label='hidden_3'))

        self.skipconnect = nn.InputChoice(n_candidates=2)

        self.fc1 = nn.Linear(neural_tuning*2, 50)
        self.fc2 = nn.Linear(50, 1)
        self.activate = nn.GELU()
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):
        x = torch.squeeze(x, dim = 0) 
        # print('1{}'.format(x.shape), flush=True)
        x,_ = self.sofa_network_lstm1(x)  
        #x = self.sofa_network_dropout(x)
        # print('2{}'.format(x.shape), flush=True)
        x,_ = self.sofa_network_lstm2(x)
        # print('3{}'.format(x.shape), flush=True)
        x,_ = self.sofa_network_lstm3(x)
        # print('{}'.format(x.shape), flush=True)
        x,_ = self.sofa_network_lstm2(x)  
        # print('4{}'.format(x.shape), flush=True)
        #x = x.contiguous().view(-1, 25600)

        output = self.fc1(x)
        output = self.fc2(output)
        output = self.activate(output)
        output = self.softmax(output)

        return output

model = Net()

import torch
from utils import accuracy
from torchvision import transforms
from torchvision.datasets import CIFAR10
from nni.retiarii.oneshot.pytorch import DartsTrainer

import numpy as np
import pandas as pd
import torch.utils.data as Data

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
#optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

# transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
# train_dataset = CIFAR10(root="./data", train=True, download=True, transform=transform)

train_dir = '/workspace/sepsis/pytorch_ver/preprocess/train_custom_age.csv'
x_train = pd.read_csv(train_dir)
input_batch, target_batch = get_data(x_train)
dataset = Data.TensorDataset(input_batch, target_batch)
train_dataset = Data.DataLoader(dataset, 16, True)

trainer = DartsTrainer(
    model=model,
    loss=criterion,
    metrics=lambda output, 
    target: accuracy(output, target),
    optimizer=optimizer,
    num_epochs=3,
    dataset=train_dataset
    )

trainer.fit()

[ultmaster]

If you are using DartsTrainer, be noted that LayerChoice and InputChoice are only supported types of DARTS. You can't use ValueChoice in you model.

[CYH4157]

Thank you very much for your answer. It makes me understand that LSTM can't use one-shot search strategy now. I have one more question. According to your experience, can one-shot search strategy be used for RNN model?

[ultmaster]

The latest version on master can. DartsTrainer can not.

[CYH4157]

Sorry for my ignorance, can you please tell me which one-shot search strategy is suitable in NNI?

[ultmaster]

What do you mean by "suitable"?

[CYH4157]

I mean which one-shot search strategy can apply to RNN model?