Closed Jiaxin-Wen closed 10 months ago
I fix this error by deleting the previously generated {exp_name}_grads.mmap
file, which possibly has some mistakes.
But I still wonder when {exp_name}_scores
is calculated :)
Hi @Jiaxin-Wen,
TLDR: the {exp_name}_scores
memmap gets created in start_scoring_checkpoint
(line 514 as of now):
https://github.com/MadryLab/trak/blob/76b13ca55f1a16a243a23aba312cf6aad57b84d0/trak/traker.py#L489-L514
In a bit more detail, after some bookkeeping, init_experiment
calls load_current_store
:
https://github.com/MadryLab/trak/blob/76b13ca55f1a16a243a23aba312cf6aad57b84d0/trak/savers.py#L359-L361
which then creates the memmap:
https://github.com/MadryLab/trak/blob/76b13ca55f1a16a243a23aba312cf6aad57b84d0/trak/savers.py#L427-L442
On a separate note, you're probably getting
File "trak/trak/traker.py", line 632, in finalize_scores
_scores_on_cpu = ch.zeros(*_scores_mmap.shape, device="cpu")
AttributeError: 'NoneType' object has no attribute 'shape'
because of https://github.com/MadryLab/trak/blob/76b13ca55f1a16a243a23aba312cf6aad57b84d0/trak/savers.py#L372-L374
This was added as an easy way to allow for scenarios where a user wants to load the final TRAK features (features.mmap
) when intermediate info (grads.mmap
) has been deleted. A downside is that it leaves users with a less interpretable error when they attempt scoring a non-existing experiment name. I'll add a more user-friendly error msg for those cases.
I have saved the
f"{exp_name}_grads"
value viatraker.score
https://github.com/MadryLab/trak/blob/76b13ca55f1a16a243a23aba312cf6aad57b84d0/trak/traker.py#L572However, when I continue to run
traker.finalize_scores
, it raises the following error:I inspect the implementation of
finalize_scores
, and find it usef"{exp_name}_scores"
to load the pre-calculated scores, which isNone
in my case.https://github.com/MadryLab/trak/blob/76b13ca55f1a16a243a23aba312cf6aad57b84d0/trak/traker.py#L627
I search the code base and still don't know when
f"{exp_name}_scores"
is calculated and saved.I believe this is not a typo haha, could you please provide any hints on this error?