ensemble

[ZhangBY-Lab]

I've attempted to ensemble the models trained using your code and my own strategy, which have scores of lb 0.724 and 0.722, but I encountered some difficulties during your inference process. May I ask if you will be updating anything related to model ensembling in your code?

[tubo213]

@ZhangBY-Lab

There are no plans to add ensemble codes, but the policy for implementation is clear.

1. inference to store keys and preds for each model. https://github.com/tubo213/kaggle-child-mind-institute-detect-sleep-states/blob/870a474449df8ea294b4a75d8fa9bd527940f6b0/run/inference.py#L122
2. Weighted average of preds per keys The output of my model is the probability of an event occurring at each step, so there is no need to adjust shape.

If you can tell me about the problems that occurred in your "inference", I may be able to provide a more detailed explanation.

[ZhangBY-Lab]

` def inference( duration: int, loader: DataLoader, models: list[BaseModel], device: torch.device, use_amp ) -> tuple[list[str], np.ndarray]:

Initialize predictions and keys

preds_accumulated = None
keys = None

# Loop through each model
for model in models:
    model = model.to(device)
    model.eval()

    # Initialize a temporary list to store predictions for the current model
    preds_current_model = []

    # Inference
    for batch in tqdm(loader, desc=f"inference {model}"):
        with torch.no_grad():
            with torch.cuda.amp.autocast(enabled=use_amp):
                x = batch["feature"].to(device)
                output = model.predict(x, org_duration=duration)
            if output.preds is None:
                raise ValueError("output.preds is None")
            else:
                preds_current_model.append(output.preds.detach().cpu().numpy())
                if keys is None:
                    keys = batch["key"]

    preds_current_model = np.concatenate(preds_current_model)

    if preds_accumulated is None:
        preds_accumulated = preds_current_model
    else:
        preds_accumulated += preds_current_model

preds_average = preds_accumulated / len(models)

return keys, preds_average

`

Here's my modified infer function; it seems to have successfully generated the submission locally, but when I submit it, I get a "Submission Scoring Error". At first, I thought it was due to the threshold, but even after adjusting the threshold, the error persists.

[tubo213]

@ZhangBY-Lab There is a problem with the keys being initialized for each iteration of the "inference loop". Please refer to my inference function to create keys.

[ZhangBY-Lab]

`def inference(duration: int, loader: DataLoader, models: list[BaseModel], device: torch.device, use_amp ) -> tuple[list[str], np.ndarray]:

Initialize predictions

preds_accumulated = None

# Initialize keys outside the model loop
keys_initialized = False
keys = None

for model in models:
    model = model.to(device)
    model.eval()

    # Initialize a temporary list to store predictions for the current model
    preds_current_model = []

    for batch in tqdm(loader, desc=f"inference {model}"):
        with torch.no_grad():
            with torch.cuda.amp.autocast(enabled=use_amp):
                x = batch["feature"].to(device)
                output = model.predict(x, org_duration=duration)
            if output.preds is None:
                raise ValueError("output.preds is None")
            else:
                preds_current_model.append(output.preds.detach().cpu().numpy())
                if not keys_initialized:
                    keys = batch["key"]
                    keys_initialized = True

    preds_current_model = np.concatenate(preds_current_model)

    if preds_accumulated is None:
        preds_accumulated = preds_current_model
    else:
        preds_accumulated += preds_current_model

preds_average = preds_accumulated / len(models)

return keys, preds_average`

Thank you for your guidance. I added a conditional statement in the loop to address the issue with the keys. Does this resolve the problem with the keys, and are there any other issues that might still exist? I will try submitting it again to see the results.

[tubo213]

@ZhangBY-Lab

The problem has not been solved. "keys" needs to be stacked.

for model in models:
    model = model.to(device)
    model.eval()

    preds = []
    keys = []
    for batch in tqdm(loader, desc="inference"):
        with torch.no_grad():
            with torch.cuda.amp.autocast(enabled=use_amp):
                x = batch["feature"].to(device)
                output = model.predict(
                    x,
                    org_duration=duration,
                )
            if output.preds is None:
                raise ValueError("output.preds is None")
            else:
                key = batch["key"]
                preds.append(output.preds.detach().cpu().numpy())
                keys.extend(key)

    preds = np.concatenate(preds)

    if preds_accumulated is None:
        preds_accumulated = preds
    else:
        preds_accumulated += preds

preds_average = preds_accumulated / len(models)
return keys, preds_average

[ZhangBY-Lab]

Wow, the submission was successful, and the score has broken through to 0.73X. Thank you!

[tubo213]

@ZhangBY-Lab Excellent! I hope you get good results.