Error when execute `finalize_scores`

[Jiaxin-Wen]

I have saved the f"{exp_name}_grads" value via traker.score https://github.com/MadryLab/trak/blob/76b13ca55f1a16a243a23aba312cf6aad57b84d0/trak/traker.py#L572

However, when I continue to run traker.finalize_scores, it raises the following error:

File "trak/trak/traker.py", line 632, in finalize_scores
    _scores_on_cpu = ch.zeros(*_scores_mmap.shape, device="cpu")
AttributeError: 'NoneType' object has no attribute 'shape'

I inspect the implementation of finalize_scores, and find it use f"{exp_name}_scores" to load the pre-calculated scores, which is None in my case.

https://github.com/MadryLab/trak/blob/76b13ca55f1a16a243a23aba312cf6aad57b84d0/trak/traker.py#L627

I search the code base and still don't know when f"{exp_name}_scores" is calculated and saved.

I believe this is not a typo haha, could you please provide any hints on this error?

[Jiaxin-Wen]

I fix this error by deleting the previously generated {exp_name}_grads.mmap file, which possibly has some mistakes. But I still wonder when {exp_name}_scores is calculated :)

[kristian-georgiev]

Hi @Jiaxin-Wen,

TLDR: the {exp_name}_scores memmap gets created in start_scoring_checkpoint (line 514 as of now): https://github.com/MadryLab/trak/blob/76b13ca55f1a16a243a23aba312cf6aad57b84d0/trak/traker.py#L489-L514

In a bit more detail, after some bookkeeping, init_experiment calls load_current_store: https://github.com/MadryLab/trak/blob/76b13ca55f1a16a243a23aba312cf6aad57b84d0/trak/savers.py#L359-L361 which then creates the memmap: https://github.com/MadryLab/trak/blob/76b13ca55f1a16a243a23aba312cf6aad57b84d0/trak/savers.py#L427-L442

On a separate note, you're probably getting

File "trak/trak/traker.py", line 632, in finalize_scores
    _scores_on_cpu = ch.zeros(*_scores_mmap.shape, device="cpu")
AttributeError: 'NoneType' object has no attribute 'shape'

because of https://github.com/MadryLab/trak/blob/76b13ca55f1a16a243a23aba312cf6aad57b84d0/trak/savers.py#L372-L374 This was added as an easy way to allow for scenarios where a user wants to load the final TRAK features (features.mmap) when intermediate info (grads.mmap) has been deleted. A downside is that it leaves users with a less interpretable error when they attempt scoring a non-existing experiment name. I'll add a more user-friendly error msg for those cases.